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ABSTRACT 

Various memoranda concerning language test 
development procedures and technical operations are compiled for 
staff at the Kuwait University Language Center from the Office of 
Tests and Measurement. The memoranda are of interest to Unit Test 
Representatives but also are intended to provide guidance to unit 
supervisors, course coordinators, and teachers in preparing 
statements of course objectives, formulating test-content 
specifications, writing and reviewing test items, and assembling 
final forms of tests. The document covers the following: job 
description of Unit Test Representatives, test production schedules, 
test development procedures, grammatical classification systems, 
test-form designation codes, behavioral objectives, item-writing 
mechanics, grammar/structure items, vocabulary items, reading 
comprehension items, items for measuring reading/listening skills, 
guide to reviewing multiple-choice questions, format, item-assembly 
procedures and form, reading/interpreting computer output, 
instructions for recordkeeping, calculating and using standard forms, 
staff assignments, and security systems for tests based upon the 
multiple-choice format and for item cards. (LB) 
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Preface 



The Test Development Manual brings together 
under one cover the more significant memoranda of 
the Office of Tests and Measurement that pertain 
to test development procedures and to the technical 
operation of that office. 

All of these memoranda are of direct concern 
to Unit Test Representatives. A number of them, 
however , are intended also to give guidance to unit 
supervisors, course coordinators and teachers in 
the preparation of statements of course objectives, 
in the formulation of test-content specif ications, 
in the writing and reviewing of test items , and in 
the assembling of final forms of tests. 

Undoubtedly, additional memoranda and revisions 
to prior memoranda will be forthcoming. For that 
reason, this manual has been assembled in loose-leaf 
form. Recipients are asked to keep their manuals 
up to date as new and revised materials are issued. 
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To Director, 0nit_Supervi8oro t Unit Tect Representative© 
Subject Job Description, Unit Tost Representative (UTR) 



1. General Nature of the Job 

A Unit Test Representative is a teacher who serves as the 
liaison person between his or her unit and the Tests and 
Measurement Office* In general a UTR is responsible for 
facilitating the implementation of T&M Office programs, 
policies and procedures within his or her unit. 

2. Appointment of UTRs and Tenure of Office 

UTRs are to be appointed by their respective Unit 
Supervisors. Appointments are normally to be for one 
academic year and are to be made no later than the last day 
of May in the immediately preceding academic year. At this 
time Supervisors are to notify the Head of the T&M Office of 
their appointees. A UTR may be appointed for additional 
years of service if this is agreeable to the UTR and the 
Head of the T&M Office. 

3. Qualifications for App ointment as UTR 

Supervisors are to attempt to appoint as UTRs individuals 
who 

a. are likely to stay or. at the Language Center for some 
time 

b. have had at least one course in testing as part of 
their professional preparation 

c. have an interest in serving as UTRs 

d. are respected by their colleagues and are capable of 
interacting diplomatically with them on controversial 
issues 
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<• Amounts of Time Expected of Tndl^i^u in Discharging 
their Duties Mn* Responsibi^ ies mm tmMt; 



Thm iMtmtt of tlM individuals will spend aaeh 
week In connection with their work as UTRs will undoubt- 
edly vary throughout the year as a conseouencm of varying 
i emends In test production schedules. Cn the average, 
however, UTRs will be expected to spend 15-2o working 
hours per week on testing matters. Supervisors are, 
therefore, to adjust the teaching loads of UTRs 
accordingly . 

5. The Accountability of PTEs 

A. At the end of e ach ■tooth in the academic year, 

beginning with S e m es ter and ending in May, each UTR 
la to prepare a report on his o* her testing 
activities during that month, copies of the report 
are to be sent to the Hcr.d of the TIM Office, 
to the UTR's Supervisor, and to the Office of the 
Language Center Dean* 

B * i fc t !l5^? nd of th * mGm ****>e year, the Head of the 
T«M Office is to evaluate the performance of each 
UTR. copies of these reports are to be sent to 
the Language Center Dean and to the Supervisor 
concerned* 

6. Specific Duties and Res ponsibility nf the UTRs 



A. 



B 



To provide guidance to the professional staff in 
his or her unit in the preparation of course and 
unit objectives and in the derivation of test 
content specifications from those objectives. 

To provide guidance to the profeeslonal staff in 
his or her unit in thm writing of test questions 
and the assembly of theme into test forms. 

c. To secure the concurrence t£ thm Unit Supervisor, 
Coordinators and teachers on test production 
schedules and to take steps necessary for subse- 
quent adherence to the schedules, 

D. To review the results of item analyses and to 
revis* or- reject- questions with undesirable 
statistical characteristics. 

E. To asaembleforms of thm pi ■ran mi test for pre- 
llmlmery administrations at thm end of the fall 
**** forms for administration in 
September of thf/ next academic year. 



F * £LZ** to lt thAt aA1 *«*tiagr materials (test 
books, answer sheets, instructions to proctors, 
ate.) needed for the September Placement Test 
administration are packaged and stored before 
thm end of the spring semester, 

G - Z° 0*i* Staff n um bers on procedures and on 

their in- 1 



BEST COPY AVAILABLE 



H. To discuss with Course Coordinators 

T&M office recommendations for changes in 
the drafts of final examinations* 



I* To prepare, if their backgrounds in measure-* 
ment and evaluations are adequate for the 
task and if they elect to do so, test 
analyses (i.e. critique based on the statis- 
tical characteristics of the tests administered 
within their units). 

J. To assist in obtaining and compiling data that 
are needed from the Units in connection with 
special studies being undertaken by the T&M 
office in cooperation with the units. 

K. To assist in implementing within the Units 
any measurement and evaluation procedures 
that have been recommended by the T&M office 
and endorsed by the L.C. Director. 

L. To assist Unit Supervisors in the administration 
of unit testing activities. 

M. To attend meetings of the UTRs that may be 
called by the Head of the T&M office. 

N. To submit the monthly activity reports specified 
in paragraph 5 A. above. 
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Donald J Malco lm 

Unit . Test Representatives £<jjr~ 

Test Production Schedules 



Early in each semester the T&M office will prepare 
and distribute to each UTR copies of a "preliminary 1 * Test 
Production Schedule (TPS) for his or her unit. This schedule 
specifies the deadline dates by which the several steps in 
the production of placement and end-of-course tests are to 
be completed. A blank copy of the TPS form for the fall 
semester of 1979 is attached to this memo. In connection 
with the TPS, the UTR is expected to do the following: 



1. Review the preliminary TPS with his or her supervisor 
and with the course coordinators to ascertain if the 
schedule is realistic from their points of view. If 
changes are recommended , the UTR is to notify the T&M 
office accordingly. The T&M office will then issue a 
revised TPS f i* it is feasible to do so. 

2. Monitor the test development activities within his or 
her unit to determine if the schedule is being adhered 
to. If there is slippage , the UTR is to notify the T&M 
office. 



Adherence to the Test Production Schedules is important for 
two major reasons: 

1. To avoid overburdening the AVA office with end-of-terro 
requests for production of tests and related materials. 

2. To avoid the crash assembly of final tests at the end 

of the term with inadequate or no reviews by unit teachers 
or by the T&K office. 



The dates given in the TPS are final dates for completion 
of the work specified. If the work can be done prior to 
these dates, so much the better. 



ERLC 



4 

1 0 



Test Production Schedule 
Fall Semester 1979 

LC Unit ' 



Preliminary | | 
Revision j j 

No* Date 



T & M OFRCE 
LANGUAGE CENTRE 
KUWAIT UNIVERSITY 



Week 



Events 



Placement Test 



Final Examinations 



Seot. 15-19 



CI 



begin, 15 th 



Sept. 21-26 



Oct, 29-3 



Oct. 6-10 



Oct. 13-17 



Oct. 20-24 



Oct. 27-31 ID-AL-ADHA 
10/29-11/2 Incl. 

Nov. 3-7 Midterms, 11/6 



Nov. 10-14 



Nov. 17-21 Al-Hijra 11/20 



Nov. 24-28 
^ 

Dec. 1-5 



Dec. 8-12 



Dec. 15-19 



Dec. 



22-26 



Xmas. 12/25 



Jan. 



Jan. 



Jan. 



29-2 



5-9 



Last Day of 
Classes 12/31 
New Year, 1/1 



LC Finals 
1/2 - 1/5 IncL 



12-16 



Jan. 19-23 Spring Vacation 
1/19 - 2/3 Incl . 



s 
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Date : Kc , bru*ry_6*__1980_ 
I'm Donald J. Hftlff??" 



MEMO 
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To Director; Assist. Director, Unit Supervisors, 

Unit Representatives, 
. Course Coordinators* 

Subject: Test Development Procedures for Final Tests 



NOTE: This meaorandua supersedes the T&M Office 
memorandum of 22nd September 1979 on the same topic. 
Please update your copy of the Teat Development Manual 
by replacing the obsolete version of TDH-3 with this 
revision* 

Outline of Procedures 

1. At the beginning of each semeater, the T&M Office 
is to send to each ELD a Unit Teat Development 
Production Schedule* This schedule specifies fhe 
deadline dates by which the Unit ±3 to coaplete 
each of the several steps in the test development 
process. Adherence to this schedule is essential 
to provide adequate time for item-writing, item 
and test reviews, and the printing of teat materials. 



ERLC 



k Smm TDM-2 in the Test Development Manual, 



8 c 



Early in the seatattr, the Unit Supervisor is to request 
each Course Coordinator to appoint a teat development 
committee for his or her course. Each of these committees 
is to consist of the Coordinator, as Chairman, and 
two or more teachers of the course. In addition, the 
UTR is to serve as an ex officio member of each TD 
committee. 

Early in the semester, the UTR is to meet with each 
Coordinator to prepare a production schedule for the. 
final examination in that course. These schedules 
are to be compatible with the Unit Test Production 
Schedule mentioned above in paragraph (1). The UTR 
is to send a copy of each of the course schedules to 
the T&M Office* 

As its first task, each TD Committee, w/ th the 

technical assistance of the UTR, is to prepare 

a preliminary 3et of content specifications for its test. 1 

These specifications are to reflect what the members 

of the committee consider to be the important instructional 

outcomes of the course* 

Following completion of the preliminary content 

specifications for the final teats within his or her 

unit, the UTR in to meet with T&M Office personnel 

to review (1) the most recently administered finaLs for 

the courses under consideration, and (2) the preliminary 

specifications for the new tests* After this review, the 

T&M Ofiice is to send a memorandum to each Courue Coordinator, 

with copies to the Unit Supervisor and UTR, cither 



The test content specifications serve as a blueprint for 
putting the teat together* Among other things, the 
specifications state which language skills are to be 
assessed and what kinds of questions are to be used in 
the assessment* For more detailed information, see 
TDM memorandum # 7* 



endorsing the specifications or recommending changes 
in them. 

6. Test Development Committee whose specifications have 
bean endorsed by the T&M Office or which accept the 
recommendations of that office are to proceed with the 
construction of their tests* If a committee finds that 
any of the recommendations of the T&H Office are 
inappropriate, the points of difference are to be 
resolved by the Courae Coordinator in consultation with 
the T&M Office. 

7. The Course Coordinator is to assign the item writing 
tasks (in accordance with the test content specifications) 
to as many of the teachers of the course as is feasible. 
By so doing, the work-load on any one teacher will be 
minimized and the talent brought to bear on the production 
of the new test will be maximized. 

8. The UTR is to see to it that item writers are provided 
with copies of the TD Office memoranda that relate to the 
types of items they have been asked to prepare. 

9. To facilitate the review of new items and their subsequent 
assembly into test forms, they are to be typed onto 

item cards in accordance with TDM-8. The UTR is to 
obtain these cards from the T&M Office and to instruct 
the teacher*, in his or her Unit in their use. 

10. All newly-prepared items are to be reviewed critically 

by the Test Development Committee for the Course for which 
they are intended. The Committee is to bo responsible 
for making decisions concerning the acceptance, rejection, 
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or modification of the items that have been submitted. 
The UTR is to be responsible for providing the oaxraittees 
with i nforma tion pertaining to the mechanics and substance 
of review procedures. (See TCM-13.) 

11. After all of the individual test items have been approved 
by the CcnmLttee, the initial type-written draft of the 
final form is to be prepared* nils draft is to include 

the cover page, the instructions to students far responding 
to the various sections and parts of the t£st, and the 
answer shee:. The UTR is to be responsible for seeing to 
it that the initial draft conforms to TIM Office conventions 
with respect to natters of format. 

12. Following carpleticn of the initial draft of the final farm, 
the Coordinator is to ask the teachers of the course who 
did not serve on the Test Development Catmittee to review 
the test individually and to subuit to him or her t heir 
recommendations for changes. In addition the Supervisor 

of the Unit is to a r r an ge to have at least, tro matters of the 
instructional staff (preferably individuals not teaching the 
course in question) take the test as a student would take 
it. The copies of the test that are to be given to these 
reviewers are not to have the " cor rec t " responses indicated 
on them. The reviewers, in addition to answering each 
question, ore to enter on the pages of the test their oooraents 
pertaining to any difficulties they encountered in taking it. 

13. The Test Development Ccnmittee is to revise its draft 

of the test in the light of the performance and garments 
of the reviewers. Copy for the final farm is then to 
be prepared. 



14. No later than the date specified in the Test 
Production Schedule, the Course Coordinator is to 
submit a copy of the test to the Unit Supervisor for 
his review. This copy is to be accompanied by the 
following documents. 

(1) The Test Content Specifications 

(2) The answer key for any multiple-choice 
questions in the test 

(3) instructions (rules and procedures) for 
reading and grading "supply-type" questions 

While a Supervisor may be critical of individual items 
in the test, the main purpose of the review at the 
Supervisor's level is to assure compatibility between 
the measurement goals of the test (as inferred from 
its contents) and the overall instructional goals of 
the Unit. If the Supervisor is dissatisfied with the 
test, he is to return it to the Course Coordinator for 
revision. Otherwise, the supervisor is to forward the 
test copy to the T*M Office for review. 

15. The T*M Office is to receive all final examinations 
of all required courses during the final week of November, 
in the case of the first semester, and during the final 
week of April, in the case of the second. Review of the 
tests by the T*M Office is to be completed by the 10th 

of the following month (December or May). The final 
examinations from all required courses of the same unit 
are to be submitted for review at the same time, to 
facilitate comparison of tests for length, level of 
difficulty, and appropriateness from the standpoint of 
skills development from one course to another. 

16. Following this review by the T*M Office, the test 
development committees will make changes as mutually 
agreed upon. If, following these revisions, the 
Supervisor finds the test acceptable, he is to attach 
a cover sheet to the test copy and forward the test to 
the T*M Office. The cover sheet is to contain the 
following inf ormation: 
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1 Identification of the course for which the 
tesat is intended. 

2. An estimate of the number of students who will 
take the test* 

3. The number of different versions (" scrambler ") 
of the test that will be required for the 
administration • 

4. Identification of the test construction committee 
and the test reviewers* 

5. A statement , signed by the Supervisor, to the 
effect that in his opinion the test constitutes 
an appropriate final test for the course for 
which it has been developed. 

The final review by the T&K Office is to be concerned 
largely with matters of test format in the interest of 
facilitating scoring and suusequent item and test analyses 
procedures. Any needed changes in format will be worked 
out by the TIM Office in cooperation with the UTR and 
Course Coordinator. 



The typing of final copy and the production of the 
required number of copies of the test and .related 
materials either within the Unit or using the facilities 
of the AVA Office are to proceed as in the past. 
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Donald J»...«Ma1nnlm , 



MEMO 



'3T 

Unit Test Representatives 



Subject Grammatical Classification System 



The attached Grammatical Classification System 
is to be used in classifying each discrete-point 
grammar/structure item that is prepared for a final 
or placement test. The classification code is to be 
entered in the upper-l^ft corner of the item card 
in the space provided. 
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GRAMMATICAL CIAS S I FI CAT I ON 
SYSTEM 

1. Simple Parts of Speech 
A. Verba 

1. Copula (BE) 

a. affirmative (SV agreement) 
b • negative 

c. Wo or Aux. in Wh. or Yes/No questions 

d. tag questions 

e. short answer 

f. past tense 

g. dummy subject 1. there 

2. it 

2. Present Tense 

a. affirmative 

b. negatives 

c. Wo or Aux. in wh or Yes/No questions 

d. tag questions 

e. short answer 

f. agreement with time 

3. Past Tense 
a-f as above 

4. Present Continuous 
a-f as above 

5. Present Perfect 
a-f as above 

6. Present Perfect Continuous 
a-f as above 

7. "Going to" future 
a-f as above 

8. -Will" future 
a-f as above 

9. a. Passive - Present Tense 

a-e as above 

9.b. Passive - Past Tense 
a-e as above 
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9.c. Passive - Present Perfect 
a-e as above 

9.d. Passive - Past Perfect 
a-e as above 

10. Past Perfect 
a-e as above 

11. Past Continuous 
a-e as above 

12. Future Continuous 
a-e as above 

13* "Used to" Past 
a-e as above 

14. Modals 

15. "had better" 

16. VP Complement 

a. 0 + infinitive 

b. to + infinitive 

B. Nouns 

1. mass or count nouns 

a. agreement with verb 

b. a little/a few 

c . some/any 

d. much/many 

2 • formation 

C. Articles 

1 . definite/indefinite 

2 . demonstrative 

D. Adjectives 

1. formation (& irregular) 

2. attributable position 

3. predicate position 
4 • comparative 

5 . superlative 

6. intensifiers (very i much) 

7. equality 
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E. 



Adverbs 



1. formation (& irregular) 

2 . comparative 

3 . superlative 

4. intenaifiers (very & much) 

5. frequency 



F. Question Words 

G. Pronouns 



II. Phrase Structures 

A. Prepositional Phrases 

1. time (in, on, at, for, since, until) 

2. place (in, on, at) 

B. Participial Phrases 

1. present 

2. past 

C. Gerund Phrases 

1. subject 

2. complement 

III. Clause Structures 

A. Connectors 

1 . co-ordinatxng 

2 . subordinating 

B. Parallelism 

C. Adjectival 

1. (who is sitting in the chair) 

2. (which I saw (it) deleted object) 

3. so that 

D. Adverbial 

1. so that 

2. time 

E. Result Clause 

F. Noun Clause 

G. Reported Speech 

1. present 

2. past 
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H* Reduced Relative Clause 
I. If clause 

!• past 

2 ♦ future 



Classification System prepared by 

Jane Gaff nay, OTR Connerce 
Steven Spur ling, OTR Arts 

Hay 1979 



DJM/St 
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To 



18 October 1978 



D J Malcolm 



fir* 



MEMO 



sutojm Test Form Designation Codes 



With the exception of informal classroom tests, each test constructed and 
adninistered within the language Centre is to be identified by an alpha- 
numeric code. This rccruir anient applies to placeman t, mid-term, and 
final tests? and to any other special-purpose tests that may be developed. 

purpose of the codft is to facilitate accurate record keeping of the 
Language Centre's testing activities ♦ 

Tly& digits and letters in tJie code represent the following in 
tha order given. 

1. Language field 

A. Arabic 

B. English 
F. French 
R. Russian 

2. Ptculty 

A. Arts 

C. Ccnmerce 

G* Engineering 
S. Science 



3. 



1. Fall 

2. Spring 

3. l—i 



4. Acadamic year 

8. 197&-79 

9. 197*-80 
0. 1980-81 
1 1981-82 



Notes The academic year Is considered to begin in Sept. The code mater 
indicate, the calender year in which the academic year commences. A teat form xs 
to be coded according to the academic year in which it la alven even thounh th- 
calendar year nay be the next higher digit. See example*. In the case of 
pretests or preliminary versions of a test (see 6) , the academic vear in which 
the final version is to be given is to be entered in the code no. even thou* 
the calendar year in which the preliminary is administered mav be different.' 
See exsspie. 



5. Type of Ttest 

P. Placement 
M. Mid-year 
F. Final 



6. Developmental Stage of Tteat 



x. Pretest 
y. Preliminary 
Z. Fine u version 



Note: The above 6 items are to be used in all test form identification 
The following 3 are to be used only if scbllcable. 

7. Alternate (parallel) form of sees test 

A. First form 

B. Second form 

C. etc 
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8. Scrambled Versions 



1. First scramble (Base form) 

2. Second scramble 

3. etc 

9. Course designation 

If a test is intended for one course f the three digit natter for that 
course is to be appended to the test code. If a test is intended for 
more than one course, the code numbers of the courses are to be appended 
in sequence with slant bars between courses. 

Examples of Test Form Designation Codes 

1. EC19PZ2 

The second scramble of the final form of the ELCJ Ccmerae placement test 
for administration in the fall semester of the 1979-80 academic vear. 

2. BC19PYA 

Form A of a preliminary version of the ELXJ Ccrnnerce placement test for 
administration in the fall semester of the 1979-80 academic year, 

Notej_ The administration date of the final form is to be criven, not that of 
the preliminary form. 

" * 3. EE18FZ 121 

The final test for ELU Engineering course 121 to be given in January 1979 
(end of first semester of 1978-79 academic year) . 

4. EE28FZ 121 

As in 3 above except that administration date is at end of sprina term in the 
1978-79 academic year. 

4. EA10MZ1 131 

Urn midterm test, first scrantole, to be given in course 13, ELU Arts, in the 
fall term of 19SO-81. 
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Teats and Measurement. Office 



Language canter 
Kuwait University 



7th January, 1978 



Memo: 




Subject: Behavioural Objectives 



As a major step in the development of the new placement 
tests, each course coordinator is to prepare a statement of 
the "behavioural objectives" of the course for which he is 
responsible. These statements are absolutely necessary if 
the placement tests are to measure the various language skills , 
abilities and aspects of knowledge that need to be taken 
into account either in assigning a student to a course at 
the appropriate level or in exempting him from that course. 

The most general purpose of any instructional course » 
including language courses, is to effect changes in the 
behaviour of the students who experience it. A change in 
behaviour has taken place when the Individual can do some- 
thing that he could not do previously or when he can do some- 
thing at higher levels of complexity or excellence than he 
could previously. A statement, then, of the behavioural 
objectives for a particular course consists of* an enumeration 
of the specific changes in student behaviours that the course 
is intended to produce. Such an enumeration is a necessary 
first-step in constructing achievement tests that are capable 
of assessing the degree to which students have acquired the 
stated competencies. 

Attached to this memorandum for your guidance is a set 
of behavioural objectives that was developed through the 
joint efforts of the TIM Office and Mr. Edward Richards, the 
099 course coordinator for ELU, Arts. Please note the follow- 
ing characteristics of this example x 

1. Specific objectives are identified under six major headings. 
These aret 



A. Speaking abilities 

B. Listening abilities 

C. Reading abilities 

D. Writing abilities 

E. (Knowledge of) Structural points 

F. (Knowledge of) Vocabulary 



The above classification of language skills and knowledge is 
a simplification of one that was formulated by John Carroll 
in connection with the design of a proficiency test in • 
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English as a foreign language . The four abilities are 
integrative in nature' the two areas of knowledge are 
discrete. This is reflected in the kinds of objectives that 
are listed under the several headings in the 099 outline. 
The ability of a person to function in each of the four 
major skill areas is conditioned by his knowledge of voca- 
bulary and of grammatical structures. For this reason, 
the structures and vocabulary that the student is expected 
to know are defined separately and in detail in the accom- 
panying set of behavioural objectives. 

2. An integral part of the behavioural. objectives concept 
is that the behaviours be observable. If they can be 
observed, they can be measured and evaluated. If they 
cannot be observed, they cannot be measured and evaluated. 
Perhaps the most: difficult task involved in the preparation 
of a set of behavioural objectives is the reduction of an 
abstraction such as "understanding" or "appreciation" to 
observable actions. Abstractions of this sort are common 
in general statements of educational objectives. 

Objective No. 1, under Listening Comprehension Abilities on 
page 2 of the attached paper, is of this type. It reads, 
"Ability to understand spoken standard English" . Such a 
statement provides no insights into either what it is that 
the student is to understand (other than it is to be in the 
spoken form) or how the student who does understand is to be 
distinguished from the one who does not. The behaviours 
listed under "Operational definitions* of "to understand" 
make it explicit, in terms of observable student responses, 
what "to understand" means at the 099 level of instruction. 
Additionally, the "Linguistic boundaries" and "Situational 
boundaries" restrict the expected student responses to these 
contexts with which he can be expected to be familiar as a 
consequence of the course and other prior experiences. 

It is .to be noted that some of the objectives given for 
099 are sufficiently specific as to require no further 
definition of terms. "The ability to ask the time and the 
ability to give the time when asked" is an example. 

In the interest of facilitating the assembly of the 

placement tests a nrocess that will require the "integration" 

of the behavioural objectives for several sequential courses 

the classification scheme described in paragraph No. 1 
above should be used, to the extent possible, in stating 



Carroll, John. "Fundamental Considerations in Testing 
for English Language Proficiency of Foreign Students" 
in Allen & Campbell, Testing English as a Second Language , 
McGraw-Hill, 1972. 



* 

The term "Operational definition" comes from the 
philosophy of science. It refers to the definition 
of a construct in terms that suggest how that 
construct is to be measured. 
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the instructional goals for each course. In some instances 
it may not be possible to classify neatly an expected 

language behaviour under one of the four primary abilities 
(for example, the ability of the student to take notes 
in English on material presented orally) . In cases of 
this sort, additional major headings will be required in 
the outline. In addition, other major headings may 
have to be introduced to encompass objectives that are 
emphasized in the course but that are not uniquely asso- 
ciated with the acquisition of English as a foreign 
language (for example: the development of study skills) . 

It will also be noted that the 099 set of behavioural 
objectives is almost silent on the matter of "expected 
fluency". This is probably a reflection of the course 
level, a level at which fluency is not a primary concern. 
Coordinators of more advanced courses may wish to specify 
the expected fluency for various language behaviours. 
These may be incorporated into the outline by addino another 
sub-category following the specifications for "Linguistic 
boundaries" and "Situational boundaries". 

The sets of behavioural objectives that you are requested 
to prepare will be more comprehensive in their coverage 
than will be the placement tests that are to be developed 
from them. In tha placement tests, for example, it will 
not be possible for logistical reasons to assess speak- 
ing ability directly. However, other uses are to be made 
in the future of the outlines that are to be assembled now. 
One use is in connection with the preparation of .end-of- 
course achievement- tests. Another use is in connection 
with the development of testing instruments for research 
purposes. 

Some concern has been expressed that putting the course 
objectives down on paper will tend to "freeze" the curri- 
culum. This is certainly not an intent of the present 
exercise. If anything, the clear specification of what 

the various courses are attempting to do now within 

units and across units should faciliate orderly revision 

in the future. 

Questions concerning the preparation of the statements 
of behavioural objectives should be directed to me at the 
T ft M Office (514330) . 
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Tests and Measurement Office 
Language Center 
Kuwait University 

4th January, 1978 

099 ArtSr Behavioural Objectives 

1. Speaking Abilities 

Obj. 1. Ability to converse with speakers of 
standard English 

A- Operational definitions 

(1) "to converse 1 * 

a. to respond appropriately to discrete 
questions 

b. to ask appropriate discrete questions 

c. to engage in a brief but meaningful 
dialogue 

(2) "Standard English" 

a* English as taught in the course 

b. English as spoken with a recognized 
educated accent 

c. English at a normal rate of delivery 

d. English with normal redundancy 

B. Linguistic boundaries 

(1) Structural: 

Expected encoding and decoding operations 
within framework of structural and 
grammatical points identified in section V. 

(2) Vocabulary: 

Limited to that specified in section VI. 

C. Situational boundaries 

Content, of conversational situations limited to: 

(1) Material that is purely factual , 
informational, or descriptive. 

(2) Material that is based on everyday 
activities and on events and places with 
which all students can reasonably be 
expected to be familiar (the weather , 
dally habits, life in Kuwait). 
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(3) Material that is familiar through prior 
class experiences (readings, dialogues , 
stories) . 

Obj. 2. Ability to ask the time and the ability to give 
the time when asked. 

Obj. 3. Ability to describe orally objects and actions 
portrayed in a picture or series of pictures 
(Same linguistic and situational boundaries as 
described under 1 above) • 

Obj. 4. Ability to summarize orally material read in 

printed passages (Same linguistic and situational 
boundaries as described under I above) • 



I. Listening Comprehension Abilities 
Obj. !• .ibility to understand spoken, standard English. 



A. Operational definitions 



(1) "to understand* 



a. to carry out non-verbal actions in 
response to instructions or directions. 

b. to paraphrase simple questions and 
single sentences presented orally. 

c. to infer topic of two-or-three sentence 
dialogue 

d. to infer location where two-or-three 
sentence dialogue takes place. 

e. to answer questions based on the factual 
and informational content of a 150-word 
monologue. 

f. to answer questions based on the factual 
and informational content of a dialogue 
of no more than three utterences with no 
more than two sentences in each utterance. 



(2) "standard Bngllsh M :See I. 1A (2) 

B. Linguistic boundaries: See I. ID (1) & (2) 

C. Situational boundaries t See I. 1C 



II. Reading Abilities 

Obj. 1. To understand communications written in standard 
English. 
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Operational definitions 



(1) -to understand " 

a. to carry out non-verbal actions in 
response to printed instructions or 
directions* 

b. to select from a set of sentences the 
sentence that best answers a given 
question* 

c. to select from a set of sentences the 
sentence that best paraphrases a given 
sentence* 

d* to answer questions based on the factual 
and informational content of a reading 
passage no more than 150 words in length* 

e* to identify the antecedents of pronouns 
used in a reading passage* 

(2) "written standard English" 

Simple or compound sentences that are grammatically 
acceptable pedagogically; Standard word order — 
Noun phrase + verb phrase + adjective phrase* 

Linguistic boundaries 

(1) Structural and grammatical 

Passages for reading to bo based largely 
on points identified in section V* Other 
structures permissible when meaning can be 
Inferred from context* 

(2) Vocabulary: 

Vocabulary in passage for reading to be 
limited mainly to that identified in section 
VI. Other high- frequency words used in class 
or in supplementary materials admissible* 
Some new words may be introduced if 
students can reasonably be expected to 
understand them* 

Situational boundaries: 

The content of printed communications to be 
limited to material that is purely factual , 
informational , or* descriptive* Passages from 
newspapers , magazines, and books appropriate 
if edited for structural and lexical suitability. 
Material for eliciting "understanding H to be 
material that the student has not previously 
encountered or studied' in the same form. 
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IV. writing Abilities 

Obj. X. Ability to communicate in written English, 



A. Operational definitions 



(1) 



"to communicate" 



to write simple sentences. 



b. 



to write compound sentences consisting 
of two simple sentences joined by a 
hign-f requency conjunction* 



to write simple <jue»tions. 



d. 



to write a number of simple sentences 
in sequence on one theme 9 given a 
series of leading questions. 



C. 



Situational boundaries: 



For evaluatibn purposes f writing limited 
to topics with which ail students can be 
assumed to be familiar. 



V. Structural Points 

A high degree of proficiency is expected in ability to 
use correctly the following structures in speaking and 
writing, and to comprehend aural and written material 
based on them. 

1. Verbs 

A. Tenses of regular and irregular verbs in the 
time-referenced situations that are specified: 

(1) Simple present and present continuous to 
express 

a. timelessness (He goes to school on the bus) . 

b. limited duration (He is studying hard for 
the final examination) . 

c. the instantaneous (He is catching the ball). 

(2) Simple past to describe an event that took 
place in the past and is regarded as completed 
(He wrecked his car.) 

(3) "going to" future to express future or present 
intention (He is going to pay for the radio.) 

2 • Nouns 9 noun-substitutes 

A. Countable and uncountable nouns 
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B. Determiners with countable and uncountable nouna 

some/any much/many 

C. Dummy subject "There* 

(1) There ls/are , was/were 

(2) Distinguish from "They are* 

Adverbs and adverbial phrases 

A. Of frequency 

(1) placement in sentence 

(2) use of "ever" in questions 

(3) specific a dv e rbs of frequency 
always never \\vmm\ Mass often 

B. of time 

now yesterday tomo rrow 
morning/ afternoon/ evening/ night 
last (night, week, month, year) 

C. Of location 

here there at ( he re ) 

in (the country) abroad 

Adjectives 

Limited to those that function as attributes and 
predicates • 

No comparisons of adjectives. 
Prepositions 

Not isolated, but in prepositional phrases designating 
time and location (on Wednesday, on the table) . 

Sentences 

A. Simple with correct word order 

Sub + verb + obj + manner + place + time 

B. Simple joined by 

when until but because and 

C. In the form of, " want soaebody 

to do soawthlng." 
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7. Questions (in tenses specified in I A above) 

A. Formation by 

1. inversions (using verbs "to be", 
"to have (got)", "must" ) 

2. Question tags (using operators is/are, 
do/does , did , must ) 

3. "Wh" words ("when", "where", 
"who" — -subjwt and bbj. — -"how") 

B. Transformation of simple sentences to questions 

8 . Concord 

A. Agreement between subject and verb. 

B. Agreement in tense between question and answer. 



VI. Vocabulary 

By the end of the 099 course, students are expected to 
have as a minimum an active vocabulary consisting of the 
words listed in the back of Targets Books 1 and 2, plus 
the vocabulary of the supplementary material common 
to all 099 classes. 
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MEMO 



Fitun 



D.J. Ha lei 



To Unit Test Representatives 

Subject Test Content Specifications 

The content specifications for a test have been aptly 
described as the "blueprint" for the test. Ideally the speci- 
fications should be sufficiently detailed so that if two 
individuals , or committees, work independently in constructing 
tons of the test, the resulting products will be strictly 
parallel in terms of types of questions used, skills assessed, 
and ievel of difficulty. The two forms will be completely 
interchangable, and will differ only in their specific items. 

Two terms are used in the material that follows that require 
definition. These are "Part" and "Section" as applied to the 
format of the test. "Part" refers to a major subdivision of the 
test. It is usually characterized by one or more of the follow- 
ing: 

(1) a printed heading in the test book and/or answer sheet 
that designates it as a major skill area (Reading Compre- 
hension, for example) j 

(2) a separate subscore that is entered in the scoring sheet 
or answer sheet? and 

(J) separate timing, either actual or suggested. Parts are 
to be identified by title and number, as Part 1, Listen- 
ing Comprehension. 

"Section" refers to a subdivision of a part. It is usually 
characterised by (1) items of the same type. 
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and (2) special instructions to the students for responding 
to that item type. Sections are to be identified alphabeti- 
cally , as Section A. In short, a test may consist of several 
major p?rts, each of which in turn consists of one or more 
sections • 

A sample set of content specifications for a fictitious 
test is attached to this memo. In connection with these , 
please note the following: 

1. The three parts of the specifications (Listening Comprehen- 
sion, Reading Comprehension and Writing) provide examples, 
respectively, of specifications involving (a) discrete, 
multiple-choice items, (b) multiple-choice items based on 
passages, and (c) free-response questions. 

2. The codes for the item types are given in the T*M Office 
Memorandum on Code Designations for Various Item Types 
(Memo No. 22) . 

3. The total number of points for each section, as given in 
the content specifications, is the maximum that is possible 
under the scoring system used in reading the paper; If 
differential weights are used in connection with arriving at 
"marks" , these should be given in the "Summary Table" at 

the end of the specifications. 
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Sample 

Test Content Specifications 



Name of Test Arts - Commerce Placement Test 

Form EAC 11PZ Administration Date: Sept, 19 81 
Prepared by; 



Part 1. Listening Comprehension 
Section A, Statements 

(1) Item type LCS-4MC 

(2) Number of items 8 

(3) Number of points per item 1 

(4) Total number of points for section 8 

(5) Stimulus material 

Simple statements that sample vocabulary , structures 
and situations defined in behavioural objectives- for 
Arts 099 4 131 and Commerce 099 and 101. One each 
appropriate for each course in each ELU . 

Section B, Dialogues 



(1) Item type LCD-4MC 

(2) Number of items 12 

(3) Number of points per item 1 

(4) Total number of points for section 12 

(5) Stimulus material 



* ?wo-person dialogues, MftF voices. 

3 each at each course level from 099 

through 103 - 133. Vocabulary, strucv ires 

and situations as defined in behavioural 

objectives . 
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Sample 



Teat Content Specifications 
(continued) 

Part 2, Reading Comprehension 

(1) Item type 

(2) Number of passages 

(3) Number of . items 

(4) Number of points per item 

(5) Total number of points 

(6) Stimulus material 

For Arts: 1 passage not to exceed 150 words 
for each course: Q9 9. 131, 132, & 133* Vocabulary , 
Structures and Situations as given in behavioural 
objectives « For Commerce: as above Q99, 101 & 103 



RCP-4MC 
8 

5 ( average ) 
1 
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Part 3, writing 

Section A, Discrete Questions 

(1) Item type WSSNP 

(2) Number of items i 0 

(3) Number of points per item 2 



(4) Total number of points per section 20 

(5) Stimulus material 

Questions to elicit single- sentence replies 
without prom pts. Cover range in Arts from 
099 throug h 132 . and in Commerce from 099 
through 10 2. see behavioural objectives 
for relevant courses > 
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Sample 

Test Content Specifications 
(continued) 



Part 3, Writing (continued) 

Section B, raragraph Writing 



(1) Item type 



WPNP 



(2) Number of items 2 

(3) Number of points *° 

(4) Total number of points for section 20 

(5) Stimulus material 

Two assigned topics , eac h requiring a snort 
paragraph in response . No prompts. Topics 
suitable for end of Arts 131 and Commerce 101 



Summary Table 



Part 


Section 


Time 


No. of 
Points 


Weight 


No. of 
Marks 


1. L.C. 


A. Statements 


30 minutes 


8 








B. Dialogues 


self-timed 


12 








Sub - Total 


by tape 


20 


2 


40 


2. R.C. 




40 minutes 
v-,imed 


40 


1 


40 


3. W. 


A. Questions 

B. Paragraph 


50 minutes 
timed 


20 
20 








Sub - Total 




40 


1 


40 


Total 
Test 




120 
minutes 


100 




120 
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M**8 

KUWAIT UNIVERSITY 

Cum MEMO 

Sabject -3.P»* Item-writing Mechanics 



_ 21 at ...March, 2973 
iUf.- — - 



f t9m P r ^ • Mal5.pl??. 




To Ma# -- Ga ?* ne y 

Mr. Hsrbolich 
Mr. Reimer 

Mr. Spurling 



1. Each discrete item is to be typed on a 12cm x 20cm 
(approximately 5" x 8") file card. Only one item is to 

HZ W , ° n each card * Blank cards are available from the 
i&n 011 ice • 

2. In the caae of a set of items based on a passage, the 
passage is to be typed on one card; and a separate card 
is to be used for each item in the set (use a paper-clip 
to hold the set together). P P P 

3. Enter the key (correct response) on the back of the item 
card so that reviewers will not have ■ acceaT~to the key 
when they read the item. 

k ' the . u PPer-right corner of the card the ELU or 

ELU's for which the item has been developed (Arts, 
Commerce, Engineering, Science or in the caae of items 
common to two EUJs Arts-Commerce, Engineering-Science). 
Also, if applicable, give the numbar(s) of the course(s) 
for which the item is appropriate. In the case of item 
sets, this information is required only on th- first card 
in the set. 

5 * 5*'!,',^ a SPl ica °l e ' in the upper-left cornor of the card 
the following content information about the item: 

A. For all structure/grammar items, the structure/gramma- 
tical point tested by the item. 

B * u° r J . all ,r e * diae COB P r en«naion items, the skill teated 
by the item. 

°* ?"\ J n « in ««*ag *nd Science items (in addition to 
A It BJ- the content claaaif ication of the item (deacrip- 
tion, observation, classification, definition or 
explanation) . 
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6* The naaea of the itea writer and of two reviewera are to 
be given on the back of the card* 



7. A eaaple card ia preeented below: 


Pronouna 


Arte - Coaaerc* 




131 - 101 


"I hare forgotten my oandwichea." 




"It doean't natter, you caa have 




A* aoM of ua 




B. aoae of oura 




C* us aoae 




D. aoae of our 




< F20NT 


Key B 




Itmm nrenared by 




Itmm reviewed by (D 




(2) 





M«/H BACK 
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KUWAIT UNIVERSITY fS&f Cojfll Idub 



T«ta 



M-9 



Dm»: 14 October 1978 

Ref : 

MEMO 

From D J Mttl ° OM jSp^" 

To Su P B ^ laa ^.' »»lt Test ftp imuuii. 

Subject Grannar/Stmcture I tans 









A grammar/structure section is to be included in each of 
the placement tests now being developed. This negnrandun 
includes examples of sane of the more casnmly used item-tomes 

Definitions of terms used in connection with nultiple-ehoic* 
questions are given on ths last page of this msooT 



General specifications for qramMr^it rueto™ ttgas 

A. All items are to be of the nultiple-choice varletv 
with four options. vanexy 

B. Although examples of a variety of item types are 
given in this memo, each ELD is to select the two 
or three item types that appear to be most generally 
applicable to assessing the granoatlcal structures 

of caxxrn to that unit. The reason for this restriction 
is to minimize the special instructions that have to be 
given for talcing the test. 

C All st ems (except in error-recognition items) are to be 
jS^oaSj' JK dlng OT **■ mtun <* **• ■tructural 



2. Item types 

A. Sentence fill-in 

Ex. Karim in Bahrain a few weeks ago. 

A. is arriving 

B. arrives 

C has arrived 
D. arrived 
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Note: The above item type is used extensively 
be c a u se of the wide variety of structures 
that con be tested. 

B. Dialogue fill-in 

Ex. "Wfe'H be ready to leave at six. " 

"Well, Jaaeela certainly to be back by 

then." 

A. must 

B. ought 

C. can 

D. should 

Note: the above type is particulafly useful in testing 
control of b*r t^ s tructu res in spoken English. 

C. Sentence Completion 

Ex. Are these the students _ 

A. you were talking about? 

B. that you were talking about them? 

C. about that you ware talking? 

D. you were talking about them? 

D. Error recognition 

Ex. The purpose of the springs are to hold the 

A B 

cover down . No error 
C D 

Note: The student's task is to identify the 
underlined portion of the sentence that 
is ungramnatical. If he believes that the 
sentence is co rr ec t as it stands, he is 
to choose the "No error" option. In a set 
of such items, some grammatical sentences 
should be given. This iten-type is used 
mainly in testing written expression. 

E. Paraphrase 

Ex. The solution has been in the tank for two hours. 

A. The solution was putting two hours ago in the tank* 

B. The solution was being in the tank for tvo hours. 

C. The solution was put in the tank two hours ago. 

D. The solution was into the tank two hours ago. 
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3. it e m writing pointers 



A. Gramnar ite ms shou ld be written so that only cm caption 

the key, is grawnaticai within the Ling uistic context of the stan. 
All other qpticns should be ungranmatical whm placed in the stem. These 
options should, houever, be grannetical in and of tl wnhn ■] ea wt» 
considered alone. 



B. An effort should be made to control the somntlc differences 

in gramnar items. That is, where poBslhle all distractors should hold 
meaning constant and vary only in graenatlcal font. 



Good; The man to the store yesterday. 

A. is walking 

B. walked 

C. was walked 

D. walks 



Bad: The nan to the store yesterday. 

A* is jumping 

B. walked 

C. was talked 

D. helps 

C. Structure/grannar items are to test the stixient's control 

of elements of structure and granraar. Hence, the vocabulary used 
is to be as siaple as possible. Similarly the readin? load should 
minimised. In other words, "the context should be at a lower 
level of linguistic ability than the actual prcbln deing tested " 



D. 



G. 



Oily one graninatical or structural point is to be +aytfrl 
in any given item. 

The stems are to be as short and as clear as possible 
Braver, words which otherwise would have to be 
repeated in each option are to be included in the stem. 

All distractors are to be reasonably attractive and plausible. 

They are to a^ear "right" to any person who doesn't know the correct 

arwaer. Suitable distractors can be derived from (a) mistakes in 

written vrork, (B) studen* answers to "supply-type" 

questions, (C) teacher's experience, and (D) contrasting analyses 

be t w een Arabic art Bglish. 

The key (correct response) is not to be ccrapicuously Icraer 
or shorter than the distractsrs. 



H. In a series of items, the key should appear about an eoual 

rambar of tines in each of the possible choice positions (A,B,C, or D) . 
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KUWAIT UNIVERSITY 
LANGUAGE CENTRE 



QUIA fj* 




M-10 



Date: 



15th. October^ . 1938 



- : r-*J x 



MEMO 



To 



From 




Subject 



Vocabulary Items 



A vocabulary section is to be included in each of 
the placement tests now being developed* This memorandum 
contains examples of appropriate item types and "pointers M 
for writing and reviewing such items* Definitions of terms 
used in connection with multiple-choice questions are given 
on the last page of this memo* 



1. General specifications for vocabulary items 

A. All items are to be of the multiple-choice 
variety with four options* 

B. All items are to test vocabulary in context* 
However # definitions and examples may be used 
as stems* Discrete words or phrases followed 
by synonyms or paraphrases as options are not 
to be used* 

2. Item types 

A. Multiple-choice fill-in 



The student is to select the word or phrase 
that logically completes the sentence* 



Ex* 1* The dog 



at the stranger. 



A* roared 



B. barked 



C . squeaked 

D . pecked 
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Ex. 2. The specimen was dried In the 
for twenty four hours 



A. deelccat eg 

B. vibrator 

C. persa i ii Isi ni 
0. purifier 



Ex. 1 (Cndssttrable) 

Km thief was apprehends* at the scene of the crime 

(B) arrested 

(C) caught 

(D) hurt 

Ex. 2 Preferred) 



torn thief was caught: at the scene of the crime. 

(A) p r o s e cu ted 

(B) apprehended 

(C) victimised 

(D) Incarcerated 



Underlined weeds or phrases 

The etudent is to choose from the options the word 
or phrase that Is nearest in meanlna to the inkri^ 
or phrase in the mom. in pi^Sce! thar^T W * rUj)md ^ 



to be two variation* of this item type, in cm th* 
•Trivi^ at th. corr^ l S^ b Sr^rS^^ 



Bel. (Dlstrsctors fit grss«stically but not logically) 

ttwlr fuooM cm 
valuable assistance 



thsir success cans about as a result of your 



A. without 

B. in spits of 

C. because of 

D. before 
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Ex. 2. (Distractors fit logically and grammtically) 

Ha tilsonvered a new route throuoh the ncuntaina 

A. wanted 

B. found 

C. travelled 

D. captured 

C. Definition 



Ex: The inatnwwt used to measure air pressure is called a 

A. c hr onom et er 

B. hydroneter 

C. barometer 

D. hygrometer 

D. Exasples 

Ex: Wiich of the following is a malleable substance? 



A. Sugar 

B. Hydrochloric acid 

C. Rubber 

D. Copper 



Item-writing pointers 

A. Vocufculaxy itans should test vocabularly. Therefore they 
should avoid granroatical structures that stutaits mav fiixl 
difficult to ccnprehotf. 

B. All options should be grammatically coiie c L and should be pragmatically 
c o rr e ct when inserted in the stan. 

C. All options should belong to the saw word class as 
the word in the stan. 

D. T he correct option and the distractors should be at 
aHttoxiaately the same level of difficulty, (if the correct 
option is more difficult than the distractora, the answer may be 
arrived at through elimination, if the c orrect opticxi is easier, 
the better strtents may be misled.) ' 

E. Avoid using pairs of opposites or syimyms as distractors. 
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To SuperTia ora, Taache ra and Unit Taat Rapraaantatiyaa 
Sabject R.ading Coaprahenaion If aa 

Th. aaaorandua containa auggaationa for writing if a. 
(quaationa) to ..aaura raading coaprah.naion akillal. It ia 
intended for ataff oa.bara who will prapar. or r.Ti.w raading 
coaprahenaion it.u for th. naw pl.c.a.»t%..t«. Thl augga" 
tiona, however, ara alao applicabl. to and-of-couraa fata. 

Raading Coaprohonaion Skills 

To coaprah.nd a writ tan paaaaga, a atudant auat ba abla 

at iSTworoa ( 2n iTSS* "T!? ***** to * ^ P^fe 
of the worda in it, (2) eopa with tha graaaatical fora. it 

containa and (3) raapond correctly to tha graphic ayJbola 

Swaiar^rth.'!^:?.*; T?*' °* co^l^Soa. 

How.w, at tha levela for which tha plaeeaant taata ara 

S^'al^JKitaV* th * ' tUd,nt t0 " MO * wSS. ahould 

not ba a aignificant factor in d.t.raining him fat acora. 



«rint!J i il.S?r i ? d8 * ? £ r oc * bu1 -^. graaaatical atructura. and 

!r^;Ja2T!^ 1 ? r ** i L d : t * r ^ a * r ** dia « «o«p.t.nc. ia th. 

•leaentary and intaraadiata phaaaa of language develonaant th« 
fating of -pacific abxlitiaJ in th... ar^aH-uaaTrb. con. 

!?) JSf ff,??!!*!! *: **°""«P*ioiia to thi. g.a.raliration ara 
tnroali aaSatSLi? d, , tar ^ B V th r-««i«« of unfaailiar word. 



T.raa used in connection with question, ara 
defined on paga of thi a aaaorandua. 

1 

Barria, David P. Taatina Kn»i j,h a a—ond L, w ^,. ff .. 

McGraw-Hill, Inc. 1969 P 59. 
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Coapreheneion queetiona, depenoing on what the aeaeurenenx; 
aoala are, can be baaed on written paaaagee of virtuaxly any 
*en*th, froa eingie sentencee upwards. Moat likely, reading 
coaprehenaion material in the placement teata will vary froa 
single eentencea to paragrapha of not aore than about 150 words. 

Liated below are a nuaoer of coaprehenaion abilitiea that 
can be teated through aultiple-choice queationa. For siaplicity, 
the abilitiea have been atated in productive rather than in 
recognition terae. However, reaearch indicatea unaabicuoualy that 
over a wide range of talent the ability to aupply correct reaponaea 
to queationa correlate© very highly with the ability to recognize 
♦orrect reaponaea to the eaaa queationa. Soae exaaplee of iteaa are 
alao given below, but the directiona to the student have been oaxtted, 

A. Coaprehenoion abilitiea at the aingle-eentence level 

1. The ability to paraphraee a given aentencet 

Ex: George haa just returned froa hie vacation. 

A. George ia spending hia vacation at hoae. 

B. George haa juat finished hia vacation. 

C. George i3 juat about to begin his vacation. 

D. George has decided not take a vacation. 

2. The ability to renpond appropriately to a given qu*ation: 

Ex: "How does the aaount of' rain so far thia year 

coapure with that for the saae period laat year' ? 

A. "During all of laat year, two inchea.of rain foil". 

B. "The rain coaea at the saae tiae each year". 

C. "By thia tiae, there waa leaa rain laat year". 

D. "The period will be the aaaa thia year aa it 

waa laat year". 

J. The ability to aolve a aiaple problea baaed on the 
inforaation contained in a given stateaent: 

Ex: The teat waa scheduled to begin at 8.00 AM, 
but it waa delayed for fifteen ainutea. 

At what tiaa did the teat atartf 

A* Seven- forty- five in the evening 

B. Eight-fifteen in the evening 

C. Seven- forty-five in the aorning 

D. Eight-fifteen in the aorning 
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k. The ability to make a plauaible inference from 
a given statement: 



Sxi He bent ovar and duatad off hia shoes. 

A. Hia naw ahoea hurt hia feat. 

B. Ha waa about to go in tha water. 

C. He had been walking along a sandy road. 

D. Ha waa vary old. 



B. Couents on comprehension iteaa at the single-aentence level. 
1 • Advantagaa: 

- Tha reading load par item ia reduced to a 
minimum. Hence, a relatively large number 
of queationa can be presented in a given 
amount of time. This contributea to teat 
reliability. 

- The sentences can be based on a wide 
variety of topics and can represent 
different styles of writing. This contri- 
butes to test validity. 

- The single sentence haa greater flexibility 
than the paragraph in teating comprehension 
problems aaaociatad with particular 
grammatical structures. 

2 • Jiead vantages : 

- The range of comprehension skills that can 
be teatad is limited. 

- Tha absence of contextual clues may make 
tha comprehenaion of tha isolated sentence 
mora difficult than had tha sentence bean 
imbedded in a paragraph. 

- In the abaence of a context, tha atudent 
may construe a meaning that is correct 
but at variance with the meaning intended 
by tha writer of tha item. 

C. Comprehenaion abilitiaa at tha paragraph level. 

Note: Theae abilitiaa have bean liatad 
roughly in order of complexity. 

1. Tha ability to anawer informational queationa which are 
anawered directly in the paaaage in tha aaaa worda aa 
uaad in tha queation. 

2. Tha ability to answer informational queationa which 
are anawered directly in the paaaage but in worda 
other than thoaa uaad in tha queation. 





3# The ability to infer the seining of unfamiliar 
words ana phrases through the context of the 

PUM(I. 

k. The ability to identify the central taought in 
a passage in which the thought is explicitly 
stated* 

5. The ability to idontify the detail* of a passage 
that support the central thought. 

6. The ability to identify the antecedents and/or 
referents of particular words or phrases in the 
passage, 

7. The ability to identify the central theaght in 

a passage in which that thought ia not explicitly 
stated. ' 

8. The ability to «elcct an appropriate title for 
a passage. 

9. The ability to derive inferences or to sake 
deductions from the content of the passage. 
These say cover a variety of skills such as: 



a. Identifying the writer's purpose 
ov point-of-view. 

b. Recognizing the "tone" of the 
passage. 

c. ffaing information contained in 
the pasaage to solve simple 
problems. 

d. Perceiving future implications. 
Guidelines in selecting reading comprehension passages . 

1 * on which sets of reading comprehension items 

/If t0 b * baMd a *">uld not exceed approximately 150 words. 
(The sample passage at the end of this memo contains 
I5O words.) Extended passages with a large number of 
questions based on each are to be avoided because certain 
topics typically giv* an undue advantage to certain 
students and an undue disadvantage to others. In the 
interest of a valid measure of rsading ability, it is 
better to have aeveral short paragraphs representing 
different topics and styles of writing than to have one 
extended paragraph of comparable lenitth. 



In general, paragraphs selected for reading 
comprehension are to be drawn from published 
sources (newspapers, periodicals, books) and 
are to be edited so as to be lexically *nd 
structurally appropriate for the course-level 
for which they are intended. 

The passages selected are to represent good 
writing. They should be well organised, be 
logically sound, and be factually correct. 

Passages should be of such a nature as permit 
a number of questions with unambiguous answers 
that corer the range of skills called for in 
the test specifications. In general, a passage 
of about 150 words should generate at least 5 
meaningful questions. A passage of 150 words 
that yields but two questions is highly 
inefficient. 

There are no fixed rules to serve as guides in 
choosing types of reading material that are 
likely to be fertile sources of questions for 
assessing a variety of reading comprehension 
skills. Passages of the following types, however, 
seem to lend themselves well to question 
formation: 

a. Those that deal chronologically with 
a series of events. 

b. Those that give instructions for doing 
something including any rules that are 
to be applied. 

c. Those that present different' opinions 
on a given topic. 

The subject matter of the passages is to be new to the 
students. At the easiest level, the content is to be 
a new configuration of the structures and vocabulary 
with which a student at that level can reasonably be 
expected to be familiar*, At more advanced levels, the 
material dealt with should not have been encountered 
previously by the student. (The purpose of this 
requirement is to present, insofar as is possible to 
do so, the same task to each teat taker.) 

The level of difficulty of a passage intended to 
discriminate between students who are to be placed 
either in a lower-level or the next higher-level 
course covered by the placement test is to be such 
that students who have "passed" the lower-level course 
can handle the passage successfully but those who have 
failed that course cannot* 
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8. Lexical and structural elements in a passage 
are to be within the "linguistic boundaries" 
given in the teat specif icationa for pasaagea 
at a particular course level. Unfamiliar elements 
are admissible providing they are not crucial to 
understanding the passage or in answering specific 
items based on- the passage. An—exception to the 
latter point arc questions that teat the ability 
to determine meaning through context. 

Types of passages to be avoided. 

1. Passages that condense an extensive body of 
knowledge into a single paragraph 

Examples of these are 150 - 200 words summaries 
of the history of transportation or the descent 
of man. Such paragraphs, usually homemade, suffer 
from several defects. Among those defects are: 
(A) the sweeping generalizations are often over* 
statements of reality, (3) the students in varying 
degree already know something about the topic, 
(C) the generalizations do not lend themselves 
to a variety of meaningful questions, and (D) the 
questions can frequently be answered without 
reading the passage. 

2. Passages that require "outside" information 
for their full comprehension 

Examples of these are (A) a technical passage 
in engineering or science that cannot be under- 
stood without pricr knowledge of a physical law 
or theory, or (B) a description of an event that 
presupposes a knowledge of the geographical area 
in which the event occurs. 

j*. Passages that ffive an advantage to 
particular groups of students 

Examples of these are (A) passages baaed on 
topics usually covered in secondary school subjects 
that are taken by some students but not all, (B) 
passages baaed on topics relating to life experiences 
that some students but not all have had, (C) passages 
based on topics about which the members of one sex are 
likely to know more than th« raeetoorn of the other 

*tx. 

**• Passages that deal with information that is 
likely to be known by all 3tudenta in the test 
population 



Items based on such passages are usually too easy, 
too trivial, or answerable without recourse to the 
passage. 



Passages that have been taken directly 

out of the textbooks or other instructional 



materials used in specific courses 

Such passages encourage rote learning. Moreover, 
they give an advantage to those students, who have 
taken the course in which the material was included. 

F. Some guidelines for writing and reviewing questions (items) 
based on reading comprehension passages: 

1. The questions should teat significant material 
in the passages. Avoid items that are based on 
trivial detail. 

2. The question and answer choices should be 
kept as simple as possible. Reading compre- 
hension ia to be tested through the passage, 
not through the wording of the questions. 

3- Questions should not be answerable without 
reading the passage. 

The stem of the item should either be in 
the form of a question or an incomplete 
statement which is sufficiently complete 
to make clear to the student what his task 
is. The stem should set the problem. The 
student should not have to read the answer 
choices (options) to determine the problem. 
The test of adequacy of an incomplete state- 
ment is whether sufficient information is 
given to permit the formulation of 'a meaning- 
ful question. 

An inadequate stem John 



C. 



B. 



D. 



A still inadequate stem 



John went 



A. 



C. 



B. 



D. 



An adequate stem 



John went to the store because 
A. C. 



D. 
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l^ch itea on a passage l3 to be inuepenaent 

of to* other itemein that paean**, i.e. succeaa 

in anavering an ltea ia not to deoend"" on 

succesefully answering a preceding: queation. 

Similarly, the wording ot one queation in a 

aerxea is not to "give away" the answer to another 

question. 

The aeaaureaent of reading comprehenaion skills 
should not be contaminated with the measurement 
of other language skills, such as writing ability. 
The use of ••open-ended" or "supply-typ." questions 
for ascertaa.iu.ng reading coaprehenaion haa at least 
two major drawbacks. First, the reaulting score 
represents a aixtur. of reading and writing skills, 
in general, a student's reading ability will be 
underestimated if he cannot write well. Secondly, 
writing is a slow process. It is the slowest of 
the response modalities. Accordingly, the amount 
of reading material and the nuaber of questions 
that can be given in a specified period of time is 
much leaa when students are required to write out 
answers rather than to select answers. The net 

* w!**° f ^ h " a tW ° drawb * c *« in to diminish the 
validity of the test as a reading test and to 
reduce the reliability of the obtained scores. 

The following points should be observed in 
preparing or reviewing the options (possible 
answers) for each itea: 

A. There should be only one correct answer 
aaong the choices. Native speakers of 
Ingliah should have no difficulty in 
identifying he intended answer. To assure 
this, all questions and answer options 
should be "prc-tested" on native speakers 
before they are used in a final or place- 
ment test. 

B. Each option should be grammatically correct 
and should fit grammatically with the stem 
of the item. (Itom writers tend to be 
conocientious in having the stea and the 
correct answer form a grammatically accep- 
table sentence. In the case of the other 
options, they soaetiaea overlook this point. 
Consequently, students may give the correct 
reeponse not because they have understood 
the passage but because the correct response 
la the only one structurally acceptable). 

C All distractors (wrong answers) should be 
reasonably plausible to students unsure of 
the correct answer. They should reflect the 
.kindo of errors students make who do not 




have the requisite comprehension skills 
for answering the question correctly. If the 
dietractors are absurdities, students may 
arrive at the correct answer by a process of 
elimination. 

D. The right answer (the key) should not be 
longer than the incorrect answers (the dia- 
tractors). Sometimes item writers will add 
restrictive clauses to the key in. order to 
make it unambiguously correct. Test- wise 
students soon learn that the longest option 
is most likely the correct one. In. the interest 
of parsimony, all options should be as short 
as possible. If, however, the key has to be 
relatively lengthy, one of the distractors 
should be made about the same length to 
counterbalance it. 

2. As is the case with all multiple-choice 

questions, avoid using "specific determiners 11 
in the options. A specific determiner is a 
word such as "never 11 or "always" • Students 
soon learn that options containing such words 
are usually incorrect. 

F. As is the case with all multiple-choice 
questions, avoid pairs of options that are 
direct opposites in meaning. Such pairs 
provide the student with clues to the correct 
response. Either one of the pair is correct 
(which is usually the case) or both are 
incorrect. 

G. Overlapping options should not be used, 
otherwise a good student may select the first 
option that is reasonably correct and may not 
even bother to read on to find that subsequent 
options are even more correct. For example, 

a passage might state that Dave's plane was 
due in at 3:00 pm but that it was delayed more 
than 3 hours. A poor comprehension question with 
overlapping options is the following: 

Q. What time did .Dave* 8 plane arrive? 

A. After 1:00 O'clock 

B. After 3:00 0 1 clock 

C. After 6*00 O'clock 



Obviously, all of the options are correct. 
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Also to be avoided art options that are 
partially correct and partially incorrect. 
For example, a paaoage might state that 
Ibrahia went to the market to purchase some 
lamb, eggs and melons* A poor comprehension 
question with partially correct option is 
the following 

Q. What did Ibrahim purchase at the market? 

A. lamb, chicken, and fruit 

B. meat, eggs and vegetables 

C. etc. 

D. etc. 
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Sample: Reading Comprehenaion Passage 



The world ftintral wealth la running out. Even 
the United States, one of the richeat natural storehouaea 
of minerala on earth, la facing ahortagea. Between 1900 
and 1950, when ita population doubled, ita consumption of 
minerala went up toy aix times, and by 1975 that rate 
will probably double. Already, world reaerrea of auch 
essentials aa copper, lead and sine are considered 
inadequate. Aa the aearch for new aourcea goea on, it 
la clear that methods of exploration and ore extraction 
must be vaatly improved to keep coata from aoaring. But 
aa high-quality ore in Minnesota neared exhaustion a few 
yeara ago, production of iron from taconite, a hard rock 
containing ore, waa achieved. What happeaa when all the 
iron in the world la uaed up? Substitutes may be found, 
or perhaps men will be able to fish nickle-iron asteroids 
from apace to aupply earth with ore. 

1. The main topic of this passage ia 

A. the mineral wealth of the United States. 

B. new materials to replace metala. 

C. mineral ahortagea in the world. 

D. making iron from taconite* 

2. About how many times greater waa the consumption of 
minerals in the United Statea in 1975 than in 1900? 

A. Two. c. Twelve. 

Six. D. Twenty-four. 

3. The paragraph mentions the rate of lncreaae ir the 
United Statea of all tha following EXCEPT 

A. the population between 1900 and 1950. 

B. the uae of minerala between 1900 and 1950. 

C. the population between 1950 and 1975. 

D. the uae of minerala between 1950 and 1975. 

4. When the writer of the passage rates to the United Statea 
aa one of the richeat natural atorehouaea of minerala 
in the world" (line 2) , he means that the United Statea 
has 
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A, 



more money than most other countries. 



B. large supplies* of minerals that are stored in special 
buildings for future use. 

minerala in the earth than most other countries. 



c. 



D. larger supplies of minerals than it can possibly uae. 
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To which of the following does "the search for new 
sources" refer (line 8)? 

1) Locating additional supplies of minerals on earth. 

2) Finding cheap ways to use low-quality ores. 

3) Finding materials to use in place of copper, lead 
and zinc. 

4) Locating iron on asteroids close to the earth. 

A. 1 and 2 only. 

B. 3 and 4 only. 

C. If 2 and 3 only. 

D . 1, 2, 3 and 4 • 

The very high prices that concern the writer of the 
paragraph are the prices in the future for 

A. metals. 

B • exploration , 

C. ore extraction. 

D. development of substitutes. 

The production of iron from taconite is an example of 

A. the benefits that come from exploration. 

B* the increasing demand for iron in the modern world. 

C. a new method for getting iron ore. 

D. the use of a substitute for iron ore. 

Which one of the following is the most appropriate 
title for this passage? 

A. The Earth's Minerals. 

B. Metals from Minerals. 

C. Iron in Modern Technology. 

D. The Irreplaceable Ores. 
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Sub ^ cct items that are intended primarily to measure either listening 
or reading comprehension skills and those that are intended to 
measure a complex of "integrated" skills . 



Listening Comprehension items at the level with which the 
Language Centre is generally concerned consist typically of a 
taped passage or dialogue followed by a series of printed 
questions to ascertain whether the student has understood the 
oral message. Depending on the particular test, the student 
is asked to indicate his answers in one of several possible 

ways by selecting one of several printed options provided 

(multiple-choice) , by supplying a verbal answer in written 
form (supply- type) , or by giving a non-verbal response (such 
as performing certain operations on a map or table). Regardless, 
however, of the response format that is used, if the listening 
test is to be primarily concerned with assessing the ability of 
students to respond effectively to the phonology, stress and 
intonation of the spoken language, care should be taken to 
assure that as far as possible the scores that result from the 
test reflect listening comprehension skills rather than reading 
or writing skills or knowledge of vocabulary and structure. 
Some implications of this principle for the selection of appro- 
priate oral passages or dialogues and for the preparation of 
the questions based on them are these: 



1. Grammatical structures and vocabulary used in the 
stimulus material should be familiar to tha examinees as a 
consequence of their prior course experiences. If new 
structures or vocabulary appears in the passage, comprehension 
should not be dependent upon them. 

2. Printed questions based on the oral material (and the 
printed options to multiple-choice questions) should be well 
within the reading comprehension levels of the students tested. 

3. Supply type questions based on the oral material should 
require responses that are as brief as possible and that are 
well within the writing ability levels of the students tested. 

^. If non-verbal responses are used, care should be taken 

to assure that the directions for making the responses and the 
procedures for following them be as simple as possible. 

A good pretest of a new listening comprehension test 
consists of administering it as a reading comprehension test 
to a representative sample of the kinds of students for whom 
the test is intended. If most of these students do not get 
nearly periect scores on it, then successful performance on 
the test is obviously being affected by skills other than those 
associated with listening comprehension; and the test should 
be revised* 

Reading comprehension items at the Language Centre usually 
consist of a passage followed by a series of questions in 
either a mcltiple-choice or a supply-type format. If the test 
is to be concerned primarily with ascertaining the ability of 
students to comprehend printed material at the level of diffi- 
culty of the passages, then the problems of comprehension 
snould lie within the passages rather than within the quest- 
ions based upon them. An implication of this principle for 
.the test construction is that the questions (and options in a 
multiple-choice format) should be as simple as possible, re-* 
.quiring reading comprehension skills at a lower level than 
those required for comprehension of the passages. In the case 
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of supply type questions, the written responses that are 
called for should be as simple as possible and be well with- 
in the writing competencies of the students who are to take 
the test. Stated otherwise , the written responses that the 
examinee is called upon to make should be the minimal responses 
needed to indicate comprehension, or the lack thereof. If 
extended written responses are required, the resulting test 
scores will not reflect reading comprehension scores per se 
but a corabina-ion of reading and writing skills. Under certain 
circumstances, such composite , or "integrated" scores may be 
regarded as desirable. This matter is discussed in the last 
part of this memo. Under other circumstances , such as are 
considered in the next paragraph , they may not be desirable. 

One type of item that has been used in some of the LC 
final examinations in courses in which the improvement of 
both r eading and writing skills are instructional objectives 
is characterized by (1) a reading passage suitable for test- 
ing reading comprehension at the end of the course for which 
the final is intended and (2) a series of supply-type questions 
relating to the passage , directing the student to 
write his answers in complete sentences* (Prompts may or may 
not be given.) The student's score on the item is thus a 
function of the two skills. If he does not comprehend the 
passage correctly, his writing performance will be impeded 
and his writing abilities will likely be underestimated. If 
he comprehends the passage correctly but does not write well, 
his reading ability will be underestimated. In short, double- 
barreled items of this sort expose the student to "double- 
jeopardy". If the goals of a course are to develop reading 

skills and writing skills, they should insofar as possible 

be tested separately. The type of item described at the 

beginning of this paragraph can be written in such a way 
that it puts the emphasis either on the writing or on the 
reading (but not both) . If the emphasis is to be on writing 
and the resulting score is to be considered as indicative of 
writing ability, then the passage should be at a low enough 
level of difficulty so that it can be assumed that virtually 
all students for whom the test is intended can readily compre- 
hend it. Only by so doing is it reasonable to assume that 
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all students are confronted with the same task in preparing 
their responses to the open-ended questions, if , on the 
other hand, the primary purpose Q f the item is to test read- 
ing comprehension, then the passage should be at a level of 
difficulty appropriate for that purpose, but the written res- 
ponses required of the student should be kept as simple and 
short as possible. Moreover, the responses should be scored 
entirely on the basis of content rather than on the basis of 
correctness of expression. 

in some instances an "integrated" test of reading and 
writing skills or of listening and writing skills is justi- 
fiable , if the course .objectives aim directly at the develop- 
ing of skills involving the processing of input information 
and expressing the results of the processing in written form. 
An example of an integrated reading-writing test is the pre- 
paration of a short, written summary of the material contained 
in a reading passage. An example of an integrated listening- 
writing test is the writing of a short memorandum based on a 
telephone communication. In order to prevent tests of these 
types from being primarily tests of a single component, i.e 
reading comprehension, listening comprehension, or writing 
ability, the stimulus material should be well within the grasp 
of most students for whom the test is intended; and the written 
responses that are called for should involve only elements of 
vocabulary and structure with which most of the students are 
familiar. In short, the test should place emphasis on the 
integration of the basic skills rather than on the basic skills 
themselvas. The latter can be better assessed by other types 
of items. 
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Date- 15 to October, 1978 



V ^ Ref. 




KUWAIT UNIVERSITY 
Cectre 



Sabject ..? tti *«™ t 5. j&KsJataC Multiple-Choice' Iti 
Prom D.J. Halcolm 

Unit lest Representatives 



T _ Supervisors 
Teachers 



Each item written within an ELU for its new placeaent test 
is to be reviewed independently and critically by two teachers 
in that Unit other than the writer of the item. The itea review 
assignments and the administrative arrangements for conducting 
the reviews should be made by the Supervisor in consultation 
with the Unit Teat Representative. Reviewers may include the 
Unit Teat Representatives, Course Coordinators, and other teachers. 

Reviewers should read the "Pointers for Item Writing" that 
are contained in the TttH aeaoa on the writing of reading-compre- 
hension, vocabulary and grammar/structure items. They should also 
familiarize themselves with the "guide" that is given below 
and make use of it during the review process. 

The items that are to be reviewed by a particular .person 
will be given to him or her by the Unit Test Representative. 
Each itea will be on a 5 M x8" card. In the case of reading-compre- 
nension and structure/grammar items, the skill or structural point 
that the itea is intended to measure will be indicated in the 
upper-left corner of the card. The course for which the itea is 
considered to be appropriate will be designated by number in the 
upper-right corner. The key (answer) wi n be on the back of the 
card. Reviewers are to look at the key only after they have 
arrived at their own answer to the problem. 

If a reviewer is satisfied with an itea as it stands, he 
or she is to sign the back of the item card. If a reviewer is 
dissatisfied with an itea, he or she is to write his suggested 
changes (which may include scrapping the itea) on a separate 
piece of paparand clip it to the itea card. The itea cards are 
to be returned to the Unit Test Representative. The revised 
cards will aubsequeatly be aent back to the reviewer for 
approval. 
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Quid* to Seviewinie Multiple-Choice Items ' 

A. With respect to the item ma a whole , consider whether 

1. it teats effectively the skill that it is intended 
to assess. (Good vocabulary iteas test vocabulary, 
not control of structures* Good structure iteas 
test structure, not vocabulary. Qood reading 
comprehension iteas require that the passages on 
which they are based be read and coaprehended 
in order to arrive at the correct responses); 

2* there is a significantly better way to test what 
the itea tests. (A different type of itea than 
that ussd night be better suited for testing the 
language problea); 

3. it is within the appropriate range of difficulty 
for the intended test population. (Although the 
typical test should contain iteas of a broad range 
of difficulty, those that are answered by less 
than 10 percent or more than 90 percent of the 
population do not contribute significantly to 
valid aeasureaent). 

B. With respect to the stem , consider whether it 

1. poses for the examinee a clearly defined problem 
or task. (The stem should make it clear to the 
student what the problem is. He should not have 
to read the options to determine his task. Stems 
consisting of a single word are almost invariably 
undesirable); 

2. contains unnecessary information (only that infor- 
mation should be presented in the stem that io 
neoessary for answering the itea); 

3* can be worded more clearly or concisely. (The 
principle of parsimony should be adhered to 
rigorously by item writers). 

C. With respect to the options , consider whether 

1. they fit logically and grammatically with the 
stem (application of this point will vary with 



1. Adapted from "Manual of Information for Test 
Coaaittee Meabers", Test Development Division, 
Educational Testing Service, Princeton, N.J. 

2. Terms used in connection with multiple-choice 
questions are defined on the last page of thie 
memorandum. 
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the basic skill being tested. In reading compre- 
hension items, all optiona should fit tha stem 
logically and grammatically. Xn vocabulary iteaa, 
all optiona ahould fit tha ataa graaaaticallj 
but may or may not - dapanding on tha itam type - 
fit logically. In atructure/graaaar iteaa, tha 
options may fit tha ataa naithar structurally nor 
logically) ; 

2. thay can ba worded mora claarly or conciaaly 
(Vords repeated in each option can often be 
incorporated in the ataa). 

With respect to the key (the correct or best answer), 
determine 

1. which option you think is intended to be the 
correct response. (Try to answer the item before 
you refer to tha key on the back of the item card); 

2. whether the key actually anawera the queation poaed 
(This point is particularly relevant to 4 ne review 
of reading comprehension iteaa. A related point ia 
whether the keyed answer is obtainable either directly 
or indirectly from the paaaage to which it refera)| 

3. whether the key needs to be made leas obvious 
relative to other optiona or tha stem (Should it 

be made longer? Shorter? More detailed? More abatract?). 

E. With respect to the distractera . consider whether 

1. there is any possible justification for considering 
one of them an acceptable reaponae to the queation 
(There should be one and only one acceptable and 
defensible answer to a given item. Errors • commonly 
a&ade by native speakers of English should not be 
used as distractera); 

2. they are sufficiently plausible to attract 
examinees who are misinformed or inadequately 
prepared. (The diatracters must not be a set of 
irrelevant responses). 
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f-nun Donald J.. , Malco l m j^g^ 
To Unit Test RaRresentativas 

Subject standardized Cover Page for Language Center Placement Testa 



The T&H Office recommends that the attached cover page 
be used, with modifications as needed, on all Language 
center placement tests. 



R7 



DO MOT OPEN THIS TEST BOOKLET UNTIL YOU ARE TOLO TO DO SO 



EAC 10PYB 
SNl 

ENGLISH LANGUAGE TEST 
Language Canter 
English Language Units 
Collages of Arts a Cosstrct 
Kuwait University 

Directions: 



1. Write all your answers in pencil U a+UJUttp ^ nM f i r i^ J i r . . ^ i ^ ^ 

on the special Answer Sheet* Do not ♦ U^Jl fiiilf . uUoJ 

write anything In this test booklet* \ j~ KU IJka ^Lljii.K ^i,.^-^ v 



Etch question in this test Is followed ^-i J>V« J^o^Ul^l u)Ua m 
by four answers* Only one answer is Usi — jr r t - U^u^l^^l IJl* 
correct. Put an X in the box on your *Sfl u u . KP^^J ^ K Su^U^-i 
answer sheet to show the answer you . **U^I 

think is correct* Make your X very 
dark and easy to see* 
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If you are not sure of the correct answer **U W ^ !Jt m 

to a question, make the best guess that JjUs pi al+Ui - rjr rf 1 

you can, and go on to the next question. 1 1 j 1 > f Jr y qjg^ 
Erase any answer you wish to change* • ^Ul J*VJl ^1 Jfal^ 

2. This examination consists of three parts. #, >*Ia5>S ^.^^11 Ioa { t 

: J, IS W>«— 

PAST TIME 

1. Grammar 12 minutes " y,' j T — 

2. vocabulary 13 minutes <. tr <^U>J1 _ 

3. Reading Comprehension 30 minutes «« r* j jiiiiM^n _ 

3. If you finish one part of the examination ^ la** -"-J." 1 ,J1 (r 
early, Do Not go back to another part or oJ-^O 1 *W ^Ul V- 11 
begin a new part. Wait until the examiner ^ , •.. f n < ...i. J i— .n v n 
tells you when to begin a part and when . '^i 11 ^ r 1 ^fcll ■^•Jl 

to stop* 

DO NOT OPEN THIS TEST BOOKLET UNTIL YOU ARB TOLD TO DO SO 

61 



KUWAIT UNIVERSITY ^^5°' 

LANGUAGE CENTRE \lSu oU - im / J 

. fnjU ^ M , , mim 1 1- odes M-15 vr-^i 

line: 17th October^- 1919 - — • 

MEMO 

Irom Donald J. JtolC^Llm^^fV^ 
To OTRS 

Subject Test Item*- Assembly Form: Explanation of Procedures 



1. The Test Item-Assembly Form (see sample , following) 
is a record of the composition of the items making up a 
given test, as well as an indication of the degree of rel- 
ative effectiveness of each item on that test in ranking the 
subjects who took it. The TIAF, as revised , provides a cross- 
reference with the item-card file (see M-20) for the efficient 
retrieval, via the accession-number system, of an* item 
suitable for inclusion in the assembly of a later test form. 
The accession number is a hyphenated, seven-digit identifica- 
tion code located in the first column of the revised Test 
Item-Assembly Form (TIAF) and in the upper right-hand corner 
of each item card. 



2. At the top of the TIAF, information is supplied regard- 
ing the unit using the test, the test ID number,, the name of 
the part of the test (e.g., vocabulary) concerned, and the 
names of those responsible for the preparation of the testing 
and scoring materials. The bulk of the TIAF is for the entry 
of the following data: 



a. the accession number 

b. the item numbers of the base and scrambled 
(A * B) test versions 

c. the key, the origin of each item— whether 
old or new, and, if old, whether an equating 
or non-equating item 

d. the tests in which the items had previously 
been used 

e. their item numbers in those old tests 

f. a record of the items 1 relative difficulty 
on the new test as determined by the post- 
administration item and test analysis 



« fid 



g. the effectiveness of each item in ranking the 
test subjects on the part of the test (e.g., 
grammar) in which they are found, relative to 
the ranking effectiveness of that part of the 
test taken as a whole (r - bis.) • 



3. At the end of the TIAF are totals for each letter 
(A, B, C, D) keyed, to assure a sufficient distribution of 
each. The relative difficulty of that portion of the test 
is also indicated there, as is the correlation coefficient 
for that part as a whole. 

4. Except for the last two columns, the TIAF should be 
filled in at the time of test construction. Following test 
administration and then item and test analysis of the results, 

the last two columns can be filled in, One such form ultimately 

with all item accession numbers (see table by unit, in Memo-21) 
included—- should be in the T&H Office files for each norming 
and placement examination for every unit. 
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From* Donald J. Malcolm 

Toi Unit Test Representatives 



Tests and Measurement Office 
Language Center 
Kuwait University 



21st February, 1979 

Subject: Reading * Interpreting the computer Output 
for LC Item Analyses 

Below is an example of the computer output -for one test item. 

■ No. A B C D. XJCET NR FT 

1 153* 7 16 13 U 0 1,000 

76 .500 3 .500 8 O00 6 • 500 5.500 0.000 

52.051 38.002 43.583 43.831 4SJ18 0.000 

PPfJUB DELEIV R 

0.765 10.111 0.5li 



the column headings designate the foil owing r 

1. A, B, C and D identify the four options in a four* 
choice question. 

2. Omit refers to those individuals who did not respond 
to the question but who went on to answer subsequent 
questions. 

3. NR is an abbreviation for "not reached". An individual 
is regarded as having not reached a particular question 
If he does not answer that question or any subsequent 
quest ions • 

PT is an abbreviation- for "proportion of all students 

who attempted the question"* This proportion is equal 

to N - NR , where N is the total number of students tested. 

N 

5. P Plus (p+) refers to the proportion of individuals 

who attempted the question and who answered, it correctly* 
Note that P+ is the same as the proportion of all 
individuals tested who obtained the correct answer only 
in the case of items that have been attempted by every- 
one. P+ is equal to N (Correct) . 

N-NR 

P<*--ls used as an index of item difficulty. The more 
difficult. ±he item the smaller the proportion of 
individuals who answer it correctly. However, there 
arm two major disadvantages In using P+ in test development 
work.. First, the scale runs tho wrong way. Secondly, 
torn. scale is not an equal interval scale, i.e., a 
difference in P+ of .05 near the ends of the scale 
represents a greater difference in Item difficulty than 
acdlfferencc of .05 near the middle of the scale. Hence , 
values of P+ cannot be legitimately averaged. Because 
of these limitations, delta f>\i is used as the index 
of Item difficulty. 
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6. Delta ranges in value from about 5.0 for very easy 

. items to about 21*0 for very difficult ones* Delta 
values are related to P+ values through the normal 
curve. Values of P+ corresponding to values of 
delta are given in Appendix I. 

7. R designates r-biaerial, the coefficient of correlation 
between answering a question correctly and total score 
on the part of the test in which the question appears* 
It is a measure of the ability of an item to distinguish 
(or discriminate) between the better students (those 
who do well on tha test as a whole) and the poorer 
students. Additional information pertaining to r-bia 

is given in Appendix I. 

Under the columns headed A, B, C, D, Omit and NR, three 
rows of figures are given. These figures convey t:he 
following information: 

Row 1. The numbers of students (frequencies) who chose 
responses A, 3, C, or D or who did not attempt 
or reach the item. A+B+C+D+OMIT+NR - total number 
of individuals tested. The correct response to 
the item is designated by an asterisk <*) . 

row 2. The percentages of all the students tested who 
chose responses A, B, C, or D, or who omitted 
or who did not reach the item. These percentages 
correspond to the frequencies given in row 1. 

Row 3. The mean scores on the part of the test in which 
the item is Included of those students who chose 
responses A, B, C or D, or who omitted or who 
did not reach the item. In arriving at this mean 
scorer each student's raw score on the part of the 
test (the number of questions answered correctly) 
is converted to a standard score in a score 
distribution with a mean of 50 and a standard deviation 
of 10. The resulting standard scores will range 
range from a low of about 30 to a high of aBout 
70. A mean score of about 50 indicates that the 
group of students is , as a whole, above average 
in ability. 

Example of Interpretation of Item Analysis Data 

The data given on page 1 may be Interpreted as follows t 

Some 153 of the students tested chose option A, the correct 
answer to the item (indicated by the asterisk after 153). 
Some 7, 16, and 13 students, respectively, chose distractors 
B, C, or 0. A total of 11 students skipped the item. All 
200 students, however, went on to answer subsequent questions. 
Hence, no individuals can be considered not to have reached 
the item <N*K>) and all 200 students can be regarded as 
having attempted the item <p+ - 1.00) • The mean part score 
for the students who answered correctly was 52.051. Since 
this mean is above 50.000, it can be concluded that the 
students who chose the correct response were on the average 
superior (in terms of abilities measured by this part of the 
test) to those who chose the other options or who omitted the 
item. The mean scores of students who answered Incorrectly 



or who omitted the item are all below SO.OOO f indicating 
that these students were lower in ability than those who 
answered correctly. 

Slightly more than three-quarters of the 200 students tested 
answered this question correctly (P+ = 0.765). This value 
of P+ corresponds to a delta of 10.111. The biserial-r for 
this item, 0.511, indicates a satisfactorily high correlation 
between answering these questions correctly and overall per- 
formance on the part of the test in which the question 
appeared. 

The item analysis data for this item indicates that it is 
a relatively easy item that is functioning well. The better 
students tend to answer it correctly and the poorer students 
are lured away by the distractors. All distractors are 
functioning and none prove more attractive to better than 
to poorer students. 
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Appendix 



Item Deltas and Blaerial-r's 



The most straight forward measure of the difficulty of an 
item is simply the proportion of students who attempt the 
item and get the right answer <p+) • These proportions are 
reported in the analyses that are run on the Language 
Center tests. However, there are two undesirable features 
of p+ as a measure of item difficulty. One is that the scale 
of proportions rune in the opposite direction to that of 
difficulty (the eaey items have larger p+ values than the 
harder ones). A more serious weakness of p+ is that the 
scale is not an equal interval scale. A difference between 
two p+ values of .10 near the center of the scale (say 
between .40 and .50) represents a smaller difference in the 
actual difficulties of the items than does the same differance 
at the ends of the distribution (say between .80 and .90). 
Because the p+ values do not constitute an equal- interval 
scale f mean p+'s and standard deviations of p+' s for a set 
of items cannot logically be calculated. To avoid these 
two limitations of p+, the delta scale (^) of item 
difficulty is often used for test development work. Delta 
is related to p+ through the normal curve. Tho table below 
gives the values of pt that correspond to values of delta. 



Relationship between values of and p+ 



very easy items < 6 



.98 
.96 
.93 



easy items 



.89 
,84 

.77 



middle dif f 
iculty iti 



.69 
.60 
.50 
.40 
.31 



hard it 



.23 
.16 
.11 



10 




very hard 
items 



.07 
.04 
.02 



A second important: statistic that is obtained for every item 
in an item analysis is r-bis, which is short for biseriai 
r— a variety of correlation coefficient. Its possible 
values run. from -1*00 through 0.0O to +1.00. it is a 
measure of the relationship between the scores that individuals 
receive on the single item (either a "1" for correct or a 
"0" for incorrect) and the total scores that the same 
individuals receive on the. part of the test in.. which the 
item appears. If , say, 20 per cent of the students get the 
item correct and if these same students have the top 20 per 
cent of the scores on the part of the test in which the item 
is included , the relationship between answering the item 
correctly and performance on the part of the test as a whole 
is very high. Such an. item is said to "discriminate* 4 very 
well between the students with the higher part scores and 
those with the lower scores. In such a case as this, the 
r-bis will be very close to its maximum value of 1.00 (a value 
never attained in practice) . If the students who "pass" the 
item do no better on the test as a whole than those students 
who miss the item, then the item has failed to separate the 
better students from the poorer. In this case the discriminating 
power is zero and the r-bis is zero. Occasionally, an item 
will have a negative r - bis. This implies that the poorer 
students rather than the better ones tend to choose the . 
"correct 14 response. Almost invariably this situation indi- 
cates a defective item. Either the item has been incorrectly 
keyed (i.e. the wrong option is specified as the right answer) 
or one or more of the distractors is for some reason lead- 
ing the better students away from the intended answer. 



An achievement test or a placement test is by definition 
a device for making discriminations among individuals with 
respect to what they have learned. Accordingly, if the 
test is to be an efficient one, each item in it should be 
capable of distinguishing to same degree the better students 
from the poorer ones. Unfortunately— from the point of 
view of ease of interpretation of the results of item 
analyses— there are no fixed rules for deciding whether the 
biserial-r for a particular item is sufficiently high to 
justify its continued use in a test or sufficiently low to 
justify its rejection. If the items in a test are very 
homogeneous with respect to the abilities they measure, the 
biseriala will run high- — saft in the • 60s and even up in 
the .70s. In a vocabulary test, for example, that is a 
sample of high-frequency words introduced in a specific course, 
the biserial-r will be high. In a vocabulary test made up 
of low-frequency words selected from different subject areas 
such as biology and literature, the biseriai will be much 
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lower because of the heterogenity of the content represented 
by the items. Reading comprehension items usually have 
lower biserials than structure items because of the wider 
variety of skills measured by the reading comprehension 
items than by the structure items* 

In general, items with biserials below .30 should be 
examined for possible flaws* It may be that one or two 
distractors are not functioning and revision is needed* 
The item analysis data will be most helpful in this 
inspection* Iteira that have low biserials and high deltas 
are usually faulty. The high delta comes about because 
the item may have a double key (two correct answers) or 
it may have no acceptable answer* Consequently few students 
choose the intended response ( hence , the high delta 
indicating a difficult item) ; and there may be no relation- 
ship between answering this question "correctly" and 
performance on the rest of the test (hence the low biserial) . 

Occasionally an item will have a very low biserial-r, but 
inspection reveals no apparent flaws in it. It may be that 
the item measures something different than what is measured 
by the other items in the test of which it is a part. 
Consequently student performance on that item may not 
correlate highly with the performance on the remaining items. 

Such an item if the skill it measures is important 

may not contribute to the internal consistancy reliability 
of the test but it will contribute to the validity of the 
test as a predictor of language performance. For this reason, 
one should guard against throwing out items without looking 
at them simply because their biserials fail below some 
arbitrary value. 
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MEMO 



Date : .28th Oct . 1JU 1272. . 

Rcf : 

Fn*n Donald J . Malcolm 

Xo Unit Teat Repreaentativea 

Subject Inatructiona for Completing the "Siiim« ry F ora = Iteg 
and Teat Analyaia Data" 



A copy of the Summary Form ia attached to thia memorandum. 
All the information required to complete thia for* ia obtainable 
from the content apecificationa for the teat and from the 
computer output for the item and teat analyaea. In making out 
the form, pleaae note the following: 

1 . Heading (page 1 ) 

A. Form Deaignation (Self-explanatory) 

B. Teat Population 

Identify the group of atudenta who took the t%eU 
For exauple, atudenta entering the College of Arta, 
or atudenta completing the 098 courae in the College 
of Engineering. 

C # Language Unit and Adminiatration Date (Self-expla- 
natory) • 

2. Teat Compoaition and Timing (page 1) 

A. Part, Name of Part, Number of Itema, Item Type, 
aad Time allowed can be obtained from the teat 
book or the teat apecificationa. In apecifying 
"Item Typea'' uae the codea giren in the T(N Memo 
on "Codea for Item Typea". 

B. The "notea" aection ahouJ.d be uaed to deacribe 
any unuaual acoring prociduree for multiple-choice 
itema (auch aa the differential weighting of 
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responses or the application of penalties for 
guessing). Alao to ba notad here ara differential 
weightings of part acores, if any, in determining 
the total acora. 

Obtained Teat Statiatica (page 1 ) 

A. Mean and S.I). (Standard Deviation). 

Theae statistics for the^part and total scores 
ara siren in the computer output directly after 
their reapeotive score distributions. 

B. Median 

The median (which is the point above which and 
below which fifty percent of the scores in a 
distribution lie) is not given in the LC computer 
output. See any eleaertary statistic book on how 
to calculate the median or aak aoaeone in the 
TUf Office to show you how to do it. 

C. Range 

The range of scores for a distribution is froa 
the lowest score obtained by anyone to the highest. 
If the lowest score anyone received on a given 
part of a test is 6 and the highest is 35% the 
range is 6-35* This a tat ia tic ia obtained by 
inspecting the score distribution reported in the 
computer output. 



The coefficient of reliability for each part of 
the test and for the total is given in the computer 
output directly after the correaponding score distri- 
bution. It is an internal consistency coefficient 
calculated on the baais of Kuder - Richardson formula 
20. 



This statistic ic the standard error of measurement. 
It is not reported in the L.C. computer output. 
However, it can ba quiokly calculated froa 



For teat acoraa near the aw/age score, it gives 
an indication of how much confidence we can have in a 
particular score. For acoraa in the middle range, 
the chances ara two out of three that the true scores 
lie between the obtained score plus or minus one SB 
The chances are. about 95 out of 100 that the 
true score lies between that obtained plus or minus 
two SE s . ' 



D. r 



xx 



E. SE 



m 
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Expected Heans for Tests of Middle Difficulty and Chan ce Score Data (page 1) 



If a multiple-choice test is constructed to be of middle difficulty for the 
group of students that will take it, the expected mean score (H^J can be 
determined from the following relationship ^ 

H * .50n + .50M/s « ,50n (1+ 1/s) 
<ft P 

where n * the number of items and s * the number of options. For a test 
made up of 40 four-choice items, H eX f?= .50 X 40 (1 + 1/4) » 25. 

This relationship is based on the assumption that if the test 1s of middle 
difficulty, students will answer one half of the questions correctly 
because they know the answers. In guessing on the remaining half, they 
will on the average guess 1/s of them correctly. 



H£ designates the mean chance score. It Is the average score that would be 
obtained on a multiple-choice test by a group of individuals who know 
absolutely nothing about the subject matter and who guess the answer to 
every question. It is equal to 1/s times the number of items, where s * 
the number of options. In a test of 40 four-choice questions, M~ * 1/4 X 
40 « 10. 



SD^ designates the standard deviation of the distribution of chance scores. 
It is equal to p q , 

where n * the number of questions, p * the probability of getting one item 
correct by guessing, and q * the probability of getting the item wrong (q * 
1 - P). 

In a test of 40 four-choice items, SD C « J 40 X .25 X .75 « 2.74. 

Accordingly, about two-thirds of all chance scores will fall within the 
range 7 to 13 (M ± 1 SD « 10.0 ± 2.74). 



D. H c + 1.65 SD fc 

This gives the score below which 95X of scores obtained purely by chance 
will lie. For the 40-1tem, four-choice test, this score is 10 + (1.65 X 
2.74) « 14.52, or 15, to round off to the nearest possible score. The fact 
that an obtained score is below 15 does not necessarily mean that that 
score is a chance one. It might be valid, but one cannot be reasonably 
certain. The possibility that any score above 15 is a chance score is most 
unlikely. 
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E. X N in Chance Range 



This is siuply the percentage of students whose scores fall within the 
chance range defined in 0 above. It is obtained from the frequency 
distributions of part and total scores in the computer printout. 



Item Statist ics: Frequency Distributions of Deltas and r-biserials (page 2) 
A. Deltas 

(1) Using the item analysis data given in the computer output, obtain 
trwf frequency distribution of deltas for each part of the test and 
enter these figures in the columns provided. 

(2) For each range of deltas specified in the left column, add the 
nu. ^rs of items in that range in each part and enter the total in 
the right-hand column. 

(3) The means and standard deviations of the delta distributions for 
the parts of the test are given in the computer output. However, 
these statistics are not given for the test as a whole. The mean 
delta for all items in the test is the weighted mean of the means 
of the parts. This is obtained by multiplying for each part the 
mean delta for that part by the number of items in it, sinning 
these products over the several parts, and dividing the resulting 
sum by the number of items. 

(4) The standard deviation of the deltas for the test as a whole will 
be calculated by the T&M Office. 

6. r-biser<als 

(The same procedures described above for deltas are to be followed in 
connection with the r-biserials. ) 

Speededness Data (page 3) 

Note: If scrambled versions of the same test form are given at a particular 
administration, the s p e e d e dness data called for should be for the base form only 
(usually designated "A," "1," or "kl ). The data should no£ be obtained from the 
computer output for the combined scrambles. The reason for this is that the 
positions of specific items in the scrambles vary. Hence the computer output for 
the combined scrambles gives an erroneous picture of the speededness 
characteristics of the test. 

A. Nunber of Items Finished by All Students 

For each part of the test, this ntnber is ascertained by inspecting the 
column labeled ~P1~ (for the proportion trying) in the item analysis for 
that part. The nunber of items finished by all students is the last Item 
for which PT « 1.00. 

B. Percentage of Students Who Finished All Items 

The percentage of students finishing, or completing, all the items in a 
part is 100 times the PT value given for the last Item in a part. If the 
PT value for the last item is .927, the percentage of students completing 
that part is 100 X .927 « 92. 7%. 



76 



c. 



Number of Items Finished by 60X of the Students 



A. 



B. 



This number is the number of the last Uem in a part that has a PT value 
greater than 0.799. 

Score Interconnections (page 3) 



The obtained intercorrelations among the part scores and between the part 
scores and total scores are given at the end of the computer output for the 
test analysis. 

Corrected for Attenuation 

A coefficient of correlation corrected for attenuation is an estimate of 
what the correlation between two sets of scores would be if each set were 
perfectly reliable. It is obtained by dividing the obtained r by the 
square root of the product of the reliabilities of the two sets of scores. 



In effect, the corrected r's adjust for differences™ the reliabilities of 
the different parts as a consequence of the different numbers of items they 
contain. 



Analysis Recommendations 

part of the report will be completed by the T&M Office and discussed with the 
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T t M Of fine 
Language Centre 

so— mbt gggtt ctm wd qgr ggxgsrg paea Kuwait uaivmty 



Ftsm Deelxaiatian Racuity 

M population Administration Data 



Teat CcEpoaltlorr and Tlalng 
Part gag of Part Noataer of Itena Item Type Tine Allowed 



TOTAL 
Notee 



Obtained test Statletlca 
Part W Mean S.D. Madlan Ffcnqe r xx SBn 




TOTAL 
Note* 



sac 



tad Magna for-Tteata of Middle 
lculty and Chance Score Data 



Part H Jap^ l\. SO^ 1^+1.653^ % N In Chance 

— — Range 



Itaa Sta t istics - Pr umj n>L V Distributions 
Deltas 



Obtained 
Deltas 

19+ 

18-18.99 

17-17.99 

16-16.99 

15-15.99 

14-14.99 

13-13.99 

12-12.99 

11-11*99 

1010.99 

9-9.99 

8.8.99 

7-7.99 

6-6.99 

5-5.99 

No. of 
it 



Part I 



Part II 



Part HI 



Part IV Part V 



Stotal 



Mean Delta 
SD of Delta 



r - biserial 



Obtained 

r - bis " 

.90.99 

.80.89 

.70.79 

.60.69 

.50.59 

.40 .49 

.30.39 

.20.29 

.10.19 

.00.09 

No. of 
It 



Part I 



r-bis 
SD of r-bis 



Part If 



Part III 



Part IV Part V 



Itotal 



7<? 



o — 

r, ;> 



Qata 



Part 1 Part II Part m Part IV Pert V 

T*Mkmr of itmm f lnutwd by «I1 strtanta 

par cant of atudants f lnlahad all lbam 
nuabar of ltaav f lnlahad by 80% of tte 
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/ 


/ 


2. 


/ 


/ 


/ 
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3. 


/ 


/ 
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/ 



Part Score Intaxcagg»i»»<~»» 

coined Qjii e dj a d far Atfcaratlon 

SE£ 1 2 3 4 5 tet 12 34 

IX i x 

2 — x 2 _ x 

3 X 3 

« X 4 

5 X 5 

Total x Total 



X 

X 



X 
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Ref : 

From Donald J. Mal colm jr^p s— ~ 
To y?Ra 

Subject Item Cards for the T&M File 

(format revised: October 1979) 



- - - : rJjH 

MEMO 



!• The item card (see sample, attached) records essential 
information about the history of usage and extent of relative 
effectiveness of a particular multiple-choice test item* Once 
an item has been altered in any way, however little, it should 
be considered an entirely new item and placed on a new card. 

2. The item card carries the following information about 
an item on its obverse side: 



a. the part of a test in which it was used (e.g., voca- 
bulary) 

b. the language unit(s) in whose exams it was used 
(e.g., English: Arts) 

c. the lowest course - level for which it can be 
considered an appropriate and representative test 
item for purposes of course-final (achievement) 
exams 

d. the item accession number, a hyphenated, seven-digit 
identification number which is unique for every item 
in the T&M Office files 

e. the history of usage of the identical item on all 
previous LC - level standardized exams for which 
statistical analysis was subsequently undertaken ' 

f. the item itself aa reproduced (or cut out) from the 
test form 
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3« The reverse side of the item card carries the following 
detailed information for each administration of an itemx 



a* its history of prior usage on various LC-level 
test forms 

b. the number of subjects to whom that exam was 
administered (N) 

c. the number of subjects who selected each option 
on the item 

d. the correct option (starred) 

e. the number of subjects who reached but did not 
attempt the item (omit) 

f. the number of subjects who did not reach the 
item (N.R. ) 

g. the percentage of subjects who attempted the 
item (P total) 

h. the number of the item on this particular test form 

i. the means on the part-scores, for this part of the 
test, of subjects who selected each of the four 
options (M*A, M.B, etc* ) 

j. the means on the part-scores, for this part of the 
test, of subjects who either omitted the item (M.O) 
or did not reach it (M. NR) 

k. the percentage of subjects who attempted the item 
and responded correctly (P+) 

1* the degree of difficulty of the item relative to 
other items in the same part of the test; 

ra. the performance of the item, in ranking subjects 

for ability, when compared with the same performance 
of that portion of the test (e«.g*, grammar) as a 
whole, of which the item is a part (r - bisj* 



All items deemed acceptable on the basis of item and test 
analyses are stored in the T&M Office iten bank, available for 
instant retrieval and insertion into LC-level exams* Once an item 
has been so determined to be effective in discriminating among 
varying levels of ability in a particular type of test, it becomes 
a permanent part of the bank, available for the construction of even 
more sensitive testing instruments* It is then available for use 
either in aiding in the diagnosis of ability-levels of incoming 
students for placement purposes, or in measuring student achievement 
in language-unit course-final exams* 
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5^ Use of the hyphenated, seven-digit acceaaion numbers 
not only assigns a unique identification code to each itea 
in the file* It alao peraita aerial ordering of the iteaa for 
purposes of ready retrieval, and thus rapid asseably of new 
teat foraa. This asaeably is facilitated via the accession 
numbers, through cross reference with the Test Itea-Aaaembly 
Forms, which constitute a detailed record of each administration 
of a given test form* 

6* The only cards in the files not to be ao numbered serially 
(because they do not contain items) or to be otherwise filled 
out with test statistics are those providing essential supple- 
mentary information basic to understanding the administration 
of the items on the other cards. Examples of such auxiliary 
"non-statistical" cards include 1) those containing reading 
comprehension passages, and 2) thoae atypical aections of the 
exams of some units, tested with a multiple-choice format, such 
as "paragraph structure" (sentence ordering, in which students 
are presented the scrambled sentences of a paragraph and must 
re-order them logically)* 
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B 


1 

C 
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A 
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M.NR 
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BASE N 


A 


B 
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LANGUAGE CENTRE 



Offcc M-21 u-y^i ot ji,.v> y 
O^ ie : October 20, 1979 _ _ : 



Rcf : - : r-*J } 

MEMO 

Fnwi Victor 



W- Mason ffi Jlbj 
j 0 Unit Test Representatives 



Subject item Accession-Number Table 

i 

1. The attached table explains the item-accession number- 
ing system that is to be used in connection with the T&M 
Office item files. The appropriate numbers are to be entered 
on all item cards and on the Test Item-Assembly Form that is 
prepared for each new test (see M-16). 



2. Each item's accession number is divided into a two- 
digit language and unit pre-code and a five-digit post-code 
serially numbering each item within a unit up to a possible 
99 , 000 items and up to 5000 items for each category. Thus r 
in the following example accession number: 

22 - 15363 



the first diqit, 2, denotes the English language oro- 
gram (1 is for Arabic and 3, French). The second digit, 2, 
indicates Commerce r the second ELU in alphabetical order 
(1 is for Arts? 3, for Engineering? 4, for Law, and 5, for Science). 

2. The entire coding system of accession numbers can be 
seen in Memo 21. In the case of each unit, provision for 
possible new types of items in the future is made under the 
category entitled "(other)". 



EE (English: Engineering) : 23-00001 23-40000 



Gr: 


23-00001 


— 


2J-05000 


LC: 


23-05001 


~~~ 


23-10000 


PS: 


23-10001 




23-15000 


RC: 


23-15001 




23-20000 


Sp: 


23-20001 




23-25000 


Vo: 


23-25001 




23-30000 


Wr: 


23-30001 




23-35000 


(other) : 


23-35001 




23-40000 


(English: 


Law) : 24-00001 24« 


Gr: 


24-00001 





24-05000 


LC: 


24-05001 




24-10000 


RC: 


24-10001 





24-15000 


Sp: 


24-15001 




24-20000 


Vo: 


24-20O01 




24-25000 


Wr: 


24-25001 





24-30000 


(other) : 


24-30001 





24-35000 


(English: 


Science) : 


25- 


-00001 — 


Gr: 


25-00001 




25-05000 


LC: 


25-05001 




25-10000 


RC: 


25-10001 




25-15000 


Sp: 


25-15001 




25-20000 


Vo: 


25-20001 




25-25000 


Wr: 


25-25001 




25-30000 


(other) : 


25-30001 




25-35000 


(French) : 


30-0OO01 




30-35000 


Gr: 


30-00001 




30-05000 


LC: 


. 30-05001 




30-10000 


RC: 


30-10001 




30-15000 


Sp: 


30-15001 




30-20000 


Vo: 


30-20001 




30-25000 


Wr: 


30-25001 




30-30000 


(other) : 


30-30001 




30-35000 
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Item Accession-Number Table 



1. The first digit of each 7 -digit number indicates the 
language: 

1: Arabic 2: English 3: French 

2. The second digit indicates the unit concerned: 

o: No differentiation 

1: Arts 

2 : CoHMrce 

3: Engineering 

4: Law 

5: Science 

3. The third through seventh digits are for serially ordering 
the items xn each program , up to a maximum of 5COO items 
per item-type (e.g., grammar, vocabulary, etc.). 



(Arabic) 



lO-OOOOl 10-35000 



Gr: 
LC: 
RC: 
Sp: 
Vo: 
Wr: 

(other) 



lO-OOOOl 10-O500O 



1O-O50O1 

10-10001 

1O-150O1 — 

10-20001 

10-25001 

10-30001 



10-1000O 
10-15000 
10-20O0O 
10-25000 
10-30O00 
10-35000 



EA 



(English: Arts) : 



21-O0O01 21-35000 



Gr: 
LC: 
RC: 
Sp: 
Vo: 
Wr: 

(other) 



21-OOO01 
21-05001 
21-10001 
21-15001 
21-20001 
21-25001 
21-30001 



21-O5O0O 
21-10000 
21-15000 
21-20000 
21-2500O 
21-30O0O 
21-35000 



EC (Knglish: Commerce) 22-00001 22-350OO 



Gr: 
LC: 
RC: 
Sp: 
Vo: 
Wr: 

(other) 



22-OCOOl 
22-05001 
22-1O001 
22-15001 
22-20001 
22-25001 
22-30001 



22-05000 
22-10000 
22-15O0O 
22-20O0O 
22-2500O 
22-30000 
22-35O0O 



Key: Gr: Grammar Vot Vocabulary 

LC: Listening 

Comprehension Wr: Writing 

RC: Reading 

Comprehension (other): other kinds of iti 

Sp: Speaking 
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KUWAIT UNIVERSITY 
LANGUAGE CENTRE 



M-22 u-^i oW-r** *r* 



Dale : October JL97_a . ; £j,Jul\ 

Rcf : DJM/st _ : 

MEMO 

Fmm Donald . J. HaicaiByjj^v-> 

To Unit Test Representatives 

Subject Code Designations for Various Item Types 

The item codes given below are to be used in the test content 
specifications and in other T*M documents in which reference 
is made to the kinds of items included in particular tests, 
in using these codes, please note the distinction that is 
made in the item-writing memo (No. 12) on "Items Primarily 
intended to Measure Listening or Reading Comprehension Skills 
and then Intended to Measure a Complex of 'Integrated' 
Skills". 

The listing that follows is not exhaustive but covers the item 
types most frequently used in LC final and placement tests. 

Terras used in describing the various item types are defined 
as follows: 

Stimulus Material : 

The printed or spoken words that are presented to the student 
and to which he is required to react in some prescribed way. 

Cognitive Task : 

The mental operations or processes that the student is expected 
to perform on the stimulus material. 

Response Modality : 

A class of overt behaviours by means of which the student conveys 
to the examiner his response to the stimulus material. Respanse 
modalities include but are not limited to (1) non-verbal acts, 
2) oral responses, (3) written responses ("supply") and 
(4) selection among options presented ("multiple-choice"). 

in designating the item codes certain principles have been 
followed. These are 

1. The first letter or two designater. the major skill area 
(for example. RC ■ reading comprehension, V « vocabulary) . 
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2 The last letter or the last combination of letters indicates 
the response modality (S = supply; 4MC = multiple choice 
with 4 options? O = response other than supply or multiple 
choice) . 

3. The middle letter or letters identify the type of stimulus 
material (for example, Q = discrete questions, G = graphical 
material) • 



Example : 



G SCP S 



I 



a grammat 4 cal structure item 
sentence completion with prompts 
supply-type responses 



ff' 
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1. Items Intended Primarily to Measure Listening Comprehension 



Stimulus Material Cognitive Task Response Modality Item Code 



A* Discrete 
sentences 



13. Discrete 
questions 

C. Dialogues 



D • Passages 



-paraphrase 
-logical 
inference 

-logical 
answer 



-answer q's 



-answer q's 



-multiple choice LCS-4MC 

-supply LCSS 

-other LCSO 

-multiple choice LCQ-4MC 

-supply LCQS 

-other LCQO 

-multiple choice LCD-4MC 

-supply LCDS 

-other LCDO 

-multiple choice LCP-4MC 

-supply LCPS 

-other LCPC 



2 . items intended primariLy to measure reading comprehension 

A.) as above for listening comprehension 
B J but with UC substituted for LC 
the i tern code 



C.Jin 

dJ 



15. Graphs 



K. Tables 



-answer q ' s 



-answer q 1 s 



-multiple choice RCG-4MC 
-supply RCGS 
-other RCGO 

-multiple choice RCT-4MC 
-supply RCTS 
-other RCTO 



3. Items included primarily to measure writing ability directly 
( all items in this category require written responses ) 



StimuLus Mater iai 



A. Discrete questions 



Discrete questions 



C. Discrete sentences 



D. Discrete sentences 



Cognitive Task 

formulate logical 
replies in complete 
sentences using 
prompts 

formulate logical 
replies in complete 
sentences without 
prompts 

formulate questions, 
using prompts, for 
which the SM constitutes 
the appropriate answers 

(as in C but without 
prompts) 



Item Code 
WSSP 



WSSNP 



WQP 



WQNP 
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Stimulus Material cognitive Task Response Modality Item Code 



E. Guide questions 
or prompts 

F* Assigned topic, 
no prompts 

G. Assigned topic 



write paragraphs 



write paragraphs 



write series of 
paragraphs 



WPP 

WPNP 

WENP 



4 # Items intended to measure vocabulary 



Stimulus Material 



Cognitive Task Response Modality Item Code 



A- Discrete 
sentences 
with gaps 

B . Discrete 
sentences 
with words 
underlined 

C. Discrete 
questions 
calling for 
examples 
related to 
word being 
tested 

D. Vocabulary 
in context 

of paragraphs 



insert word 
or words 
to complete 

give synonyms 
for under- 
lined words 



give examples 



identify 
lexical 
item defined 
or referred to 



-multiple choice 

-supply 

-other 

-multiple choice 

-supply 

-other 



-multiple 'choice 

-supply 

-other 



-multiple choice 

-supply 

-other 



VC-4MC 

ves 
vco 

VS-4MC 
VSS 

vso 



VE-4MC 

VES 

VEO 



VP-4MC 

VPS 

VPO 



5 # i terns intended to measure control of grammatical structures 



S t imulus Material 



Cognitive Task R esponse Modality Item Cede 



Discrete 
sentences 
with gaps 
and prompts 

B. Discrete 
sentences 
with gaps 
but without 
prompts 

C. Sentences 
with gaps 

in paragraphs, 
prompts provided 



insert word 
or words 
to complete 



insert word 
or words to 
complete 



insert word 
or words to 
complete 



-multiple choice 

-supply 

-other 



-multiple choice 

-supply 

-other 



-multiple choice., 

-supply 

-other 



GSCP-4MC 

GSCPS 

GCPO 



GSC-4MC 

GSCS 

GSCO 



GPP-4MC 

GPPS 

GPPO 



ERLC 



O. "i 
O i 
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Stimulus Material 



Cognitive Task 



Response Modality Item Code 



D« Sentences with 
gaps in 
paragraphs , 
no prompts 

E» Two-sentence 
dialogue with 
gap in second 
person 1 s 
contribution 



insert word 
or words to 
complete 



-multiple choice GP-4MC 
-supply GPS 
-other GPO 



treat as in A or B above 



F. - Discrete 

sentences or 
questions with 
final words 
or phrases to 
be supplied by 
student 
(sentence 
completion) 

G. Discrete 
sentences 

or questions 
with several 
words or phrases 
underlined, one 
of which may be 
grammatically 
incorrect 

(error recognition) 

H . As in G above 
but without 
underlining 



Discrete 
sentences 
given with 
transformational 
instruct ions 
(supply-type) 
or without 
such instructions 
(multiple choice) 



treat as in A or B above 



identify 
ungrammatical 
portion of 
sentences , 
if any 



identify 
ungrammatical 
portion , 
if any 



-multiple choice 

-supply 

-other 



GER-4MC 

GERS 

GERO 



-multiple choice 

-supply 

-other 



perform 

transformation 
(supply-type) 
or recognize 
correct 

transformation 
(multiple-choice) 



-multiple choice 
-supply 
-other 



(N.A.)* 

GERS 

GERO 



GT-4MS 

GTS 

GTO 



6. Items intended to measure "integrated" skills 

A - Items in which list ening and writing skills are integrated : 

Stimulus Material Cognitive Task Response Modality Item Code 



Dictated 
material 



reproduce 

dictated 

material 



-multiple chxce (N.A)* 
-supply IDS 
-other (N.A)* 



not applicable 
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Stimulus Material Cognitive Task Response Modality Item Code 



Material 

presented 

orally 



produce 
summary 



-multiple choice (N.A. ) * 
-supply IOWSS 
-other (N.A.)* 



Items in which reading comprehension and writing skills are 



integrated : 

Stimulus Material 

Reading 

passages 

followed by 

open-ended 

questions 

requiring 

answers in 

one or more 

complete 

sentences 



Cognitive Task Response Modality Item Code 
write answers 



-multiple choice 
-supply 
-other 



(N.A.)* 
IRCWS 
(N.A. ) * 



"True" cloze insert -multiple choice IC-4MC 

passages with missing -supply ICS 

every nth words 
word deleted, 
(pseudo-cloze 
passages in 
which particular 
classes of words 
have been omitted 
are to be 
classified as 
vocabulary or 
structure items 
in context) 



* not applicable 
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KUWAIT UNIVERSITY 
LANGUAGE CENTRE 




Tctfi awl 



Office 



M-23 



Date : 

Rcf : 

Fnrni 
To 

Subject 



February 26 , 1980 
DJM/st 



X2\ 



Donald J • Malcolm 



>1» 



MEMO 



Director, Assistant Director, Unit Supervisors, 

Course Coordinators « OTRs 
Addition of M-23 to Test Development Manual 



1. Please add this newest memo, entitled "Test 
Cover Sheet," to your Test Development Manual . It 
is designated M-23. 

2. The enclosed form is a sample of a suitable 
format for the cover sheet that unit supervisors are 
to use in submitting final-test copy for final review 
by the T&M Office. This format has already Incorporated 
recent suggestions that TD committees be expanded to 
include UTRs and that reviewers include instructors 
both teaching and not teaching the course for which 

a given final has been prepared. 

3. An essential part of the review process is to 
have the reviewers first take the test as a student 
would (see M-3, p. 4: No. 12). Critical comments by 
reviewers are also to be made in conjunction with this 

"pre-test" and the TD committee is then to take into 
account the results of both the pre-test and the 
reviewers' incidental critical comments. 



ERIC 



94 

ICO 



COLLEGE OF ENGINEERING 
ENGLISH LANGUAGE UNIT 
TEST COVER SHEET 



(Date) 



1. Course Identification 

2. Estimated Number of 
Students Who Will 

Take The Test 

3. Number of VERSIONS 

4. Test Development Committee Members 

a) , Chairperson 

b) 

c) 

d) , (UTR) 

e) 

f) 



5. Test Reviewers 

a. Teachers of the Course 

1) 

2) 

b. Other Teachers 

1) 

2) 



In my opinion, this trst constitutes an appropriate 
final test for the course for which it has been 
developed. 
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Memo for: ELD Supervisors , UTR's f Course Coordinators 
and Teachers : M-24 



Subject: Procedures for Calculating and Using 

Standard Scores 



1 . Some reasons for using standard scores 

Standard scores are particularly useful in 
situations in which it is necessary to combine 
several test scores in order to arrive at a single, 
composite score. Two such situations that occur 
within the ELU's are these: 

(1) the combining of scores on several unit tests 
to determine an overall score for performance 
on such tests, and 

(2) the combining of scores on several components 
(course-work, unit tests and final examination, 
for example) to determine final grades. 

The basic problem associated with simply averaging 
the scores on, say, several unit tests to obtain 
an overall average is that this procedure disregards 
differences in the means and standard deviations of 
the score distributions for the different tests. 
As a result of this, the unit test with the highest 
mean score will have a greater effect than the 
others in determining the mean of the distribution 
of composite scores. Similarly, the unit test with 
the greatest variability in obtained scores (i.e., 
with the largest standard deviation) will have a 
greater effect than the others on the variability 
of the ccrposite scores. The advantage of converting 



From: 




Date: 



May 4, 1980 
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scores on several unit tests , or on the components 
entering into the determination of final grades, 
into standard scores is that the means and standard 
deviations of the constituent score distributions are 
respectively equalized • Hence, all have an equal 
weight , or effect , in the determination of the composite 
scores* In the situation where a priori weights are 
applied (as in calculating final grades from component 
scores) , the use of standard scores assures that the 
a priori weights are truly reflected in the resulting 
distribution of composite scores* 

2. Standard scores appropriate for LC purposes 

Any obtained score distribution can be converted 
into a corresponding . standard score distribution with any mean 
and standard deviation that one cares to assume* 
For example, if a test is given that results in a score 
distribution with a mean of 90 and a standard deviation 
of 15 , the test scores can readily be converted so as 
to yield a score distribution with a mean of 50 and a 
standard deviation of 10. The transformation from 
one scale to the other has no effect on the rank- 
ordering of individuals. The person who is fifth from 
the top on the original scale will be the person fifth 
from the top on the converted scale. For Language 
Center purposes, it is recommended that obtained score 
distributions (as for the unit tests) be converted to 
standard distributions with a mean of 50 and a standard 
deviation of 10 (as in the example cited above) . 

3. Using standard scores in connection with unit tests 

A. Preliminary considerations 

(1) When standard scores are used, the effort should 
not be made to assemble tests that yield scores 
on an absolute scale from, say. 60 per cent 
correct to 100 per cent correct. A difficulty 
with this approach is that only 40 per cent 
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of the total possible marks Is used to make 
distinctions among all those (the large majority) 
who will receive passing grades. The remaining 
60 per cent of the total marks simply discriminate 
among the failures and f hence , serve no useful 
purpose. In the case of tests that are to be 
used for assigning grades , it is better to have 
the tests pitched at middle difficulty for the 
students who will take them. A test made up of 
supply-type items (essay , fill-in-the gaps, etc.) 
is of middle difficulty when the average score is 
about 50 per cent of the total possible marks. 
A test made up of multiple-choice questions with 
four options is of middle difficulty when the 
average score is about 62 per cent of the total 
number of such items. Tests of middle difficulty 
tend to distribute students over the whole range 
of possible scores , and r by so doing, permit a 
more reliable assignment of final grades than is 
possible when the test scores are compressed into 
a relatively narrow range from 60 to 100 percentage 
points. 

in the case of Unit Tests that are to be given to 
sevs *al sections of the same course and that contain 
substantial amounts of response material that must 
be subjectively graded, panel marking is much to 
be preferred over the reading of papers for a 
particular section by the teachers of that section. 
The reason for this, of course, is to reduce the 
inevitable effect of differences in the frames 
of reference and standards of individual teachers. 
Standard scores based on the performance on a Unit 
Test of all students from all sections of a course 
will not compensate for differences in the grading 
practices of individual teachers when each teacher 
grades his or her students' papers. 
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4. Calculation of Standard Scores 



The conversion of an obtained test score 
to a standard score in a distribution with a nean of 
50 and a standard deviation of 10 Involves 
only the multiplication of the obtained score by a 
constant A and adding a constant B to the resulting 
product , as is shown below t 



X SS - AX 0B + B 



where is an obtained test score 

OB 



Xg S is the corresponding standard score 

A = 10 , where SD QB is the standard 
deviation of the raw score 
distribution 



B = <5 ° ~ M ° b) ' Wherft M ° B iS ^ 

mean of the raw 

score distribution 



Example: A unit test is administered to all the 
students in all the sections of a course* The mean 
and standard deviation of the resulting raw score 
distribution are then calculated. These statistics 
may be obtained directly on many "scientific" pocket 
calculators and on the HP 97 in the TIM Office. 
Assume that the mean of the distribution < m q B ) ic • 
30*0 and that its standard deviation < SD 0B ) is fl * 2 * 
Constants A and B are then determined as shoom below: 

1 » 10 » 10 - ! -> 2 

B * <50 — M— ) - <50 - 1.22 X 30) 

SD 0B 

OB - (SO - 36.60) 

- 13*40 

Hence the conversion equation becomes 

*SS = X ' 22 X OB + 13 ' 40 
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If, in thto exaaple given above, a stude it has an 
obtained score of 20, his standard score is obtained 
by multiplying that score by 1.22 and adding 13.40 
to the product. 

X ss = 1.22 X 20 + 13.40 = 22.40 + 13.40 

= 35.80 

In calculating standard scores on individual tests, the 
standard score should be rounded off to the nearest 
whole number, i.e., 35.8 would be reported as 36 . In 
the case of means of several standard scores (as in 
determining composite final grades based on several 
components) , the mean score should be reported to the 
first decimal place. Carrying out the scores to more 
decimal places implies a degree of precision that does 
not exist in the field of language testing. 

By use of a pocket calculator that can be programmed, 
the conversion of obtained scores to standard scores 
can be done very quickly. The HP 97 in the T*M Office 
is available for use in the office for this purpose. 

5* The Range of Standard Scores 

If the distribution of obtained scores is normal, 
i*e*, the distribution curve is bell-shaped and 
symmetrical about the mean-— as is usual with tests 

that are approximately of middle difficulty the 

following percentages of standard scores will fall with- 
in the ranges indicated. 
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Table 1 



Standard Score Percentage of Scores 

Range in Range 



80 


and above 


0.13 


% 


70 


to 79 


2.15 


% 


60 


to 69 


13.59 


% 


50 


to 59 


34.13 


% 


40 


to 49 


34.13 


% 


30 


to 39 


13.59 


% 


20 


to 29 


2.15 


% 


19 


and below 


0.13 


% 



It will be noted in the above table that nearly 
all standard scores will be in the range from 20 to 80. 
Unless the test on which the scores are based was extremely 
easy or extremely difficult, very few standard scores 
(about one in a thousand) will be above 80 or below 20. 

In using standard scores care should be taken to 
label them as such so that they will not be confused 
with percentage scores. A score of 80 on the percentage 
scale is only mediocre. On the standard score scale, 
80 is extremely high. 



Averages (Means) and Weighted Averages of Standard Scores 

Averages and weighted averages for several 
standard scores are obtained in the same manner as for 
any other set of scores. Given below are the standard 
scores received by a student on three unit tests , X, Y and 

Standard Scores on Unit Tests 

Student No. 5572 J[_ 

38 42 46 
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If eacn score is to receive equal weight in a composite 
total scorer add the three scores and divide by three. 

38+42+46 

= 42,0 (Simple meerj 

3 

If for some reason, it is considered desirable to give 
the score on test 2 twice the weight of either X or Y, 
multiply each score by its weight, sum these products, 
and divide by the sum of the weights, viz., 

(1) X 38 + (1) X 42 + (2) X 46 

« 43 (weighted mean) 

4 

Assigning Standard Scores for Classwork 

In the Language Center final graces are generally based 

on three components unit tests, classwork and final 

examinations. In the case of the unit tests and final 
examinations, standard scores can be calculated directly 
from the obtained score distributions. The application 
of standard scores to student performance on classwork, 
however, represents a much different situation in that 
there are no common measure of classroom performance 
and, consequently, a much higher degree of subjectivity 
enters into grades on this variable than on those based 
on tests. Perhaps the best thing that can be done under 
these circumstances is to ask individual techers to assign 
stnadara scores to students in their sections roughly 
according to the distribution given in Table 1. In the 
case of very small sections, say of fewer than 10 students, 
considerable variations from the suggested distribution 
are to be tolerated because of the operation of chance 
factors. In the case of two sections under the same 
teacher vith say 50 students, the distribution of grades 
for classwork should approximate quite closely those 
given in Table 1. 

Calculating the Final-grade Standard Score from the 
Standard Scores of the Components 

The procedure for obtaining the weighted mean score of 
the final-grade components is the same as that described 
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in section 6 of the memorandum for sta inin g the 
weighted Btm of the standard scores for several unit 
tests . 



The example that follows is baseu on the assumption 
that performance on the unit tests is to count 40 per 
cent of ths final grade , clarwork 20 per cent and the 
final examination 40 per cent. 

Standard Scores for 
Unit Tests Classwork Finals 

Student No. 08266 45 38 40 

Inasmuch as relative weights of 40%: 20%: 40% are 
the same as 2 : 1 : 2, the weighted mean of the above 
test scores is given by 

SS (Total) « (2) X 45 4- (1) X 38 4- (2) X 4Q . 4Ufi 

5 

It is recommended that the above score be multiplied by 
lO and reported as a three-digit number ,* 416. "This 
will clearly identify the total score as such. It will 
prevent confusion with the absolute percentage scale 
from 0 to 100, and it will provide for finer distinctions 
among large numbers of students (even through the 
distinction based on the third digit may be more a 
function of chance than of reliable measurement) . 

Assigning Letter Grades Based on the Final Grade 
Standard Scores 

The principle is simple enough. Make a list of the 

final grade scores in rank oraer from highest to lowest. 

Count down from the top the number of cases corresponding 

to the percentage of students to receive A's. Then, 

from that point, continue to count down further the 

number of cases corresponding to the percentage of 

students to receive B's, etc. This procedure is satisfactory 

for small numbers of students, say SO or fewer, but it 

is a time consuming process when grades have to be 

assigned to a large number of students. 
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One method that can be used to titlBAte the "cut-off* 
points for the various letter grades is to prapara a 
fraquancy distribution such as that shown on paga 11. 
Tha staps ara as follows s 

1. If tha course consists of separate sections for 
boys and girls , determine tha percentage of boys 
(to the nearest whole percent) and the percentage 

of girls. 

2. From the sections for girls , draw a spaced sample 
(every n 1 th case) equal to the percentage of girls. 
For example, if the overall percentage of girls in 
the Course is 64%, draw 64 cases from the sections 

of girls. If there is a total enrolment of 180 girls 
select every third esse team each rtrrlnn and draw the 
remaining four cases randomly from all sections. 
Follow a similar procedure in the case of the boys. 

3. Enter the tally-marks on the interval scale as 
shown in page 11. 

4. Since each case represents one percentage point, 
count down from the top the number of cases equal 
to the percentage of A' s to be given. Repeat the 
same procedure for the other gradea. 

5. If in counting down the number of cases to receive 
a particular letter grade, you find that there are 
more cases in the bottom interval than you need, you 
can do one of two things (a) give all students in 
that interval the grade in question (which will 
increase the number of students receiving that grade 
above the number intended) or (b) assume that the 
tally marks for the interval in question ara equally distri- 
buted over that interval and estimate the cut-off jura for 

the nunber of cases you require. 

Another method that can be ua#d requires a calculator. 
First the mean and standard deviations of tha distri- 
bution of the final standardized scores are determined. 
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Then the normal curve table is used to determine the 
cut-off scores that will give the desired percentage 
of cases in the upper end of the distribution. For 
further information in this method, consult the 
Tests and Measurement Office. 
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Table 2 



ELU 



Course 



Intervals 



640 

630 
620 
610 
600 
590 
5S0 

570 
560 
550 
540 
530 
520 

510 
500 
490 
480 
470 
4b0 
450 
440 
430 

4 20 
410 
400 
390 
380 

370 
360 
360 



639 
629 
619 
609 
599 
589 

- 579 

- 569 

- 559 

- 549 

- 539 

- 529 

- 519 

- 509 

- 499 

- 489 

- 479 

- 469 

- 459 

- 449 

- 439 

429 
419 
409 

399 
389 

379 
369 



Tallies 

1 

1 
1 

11 
111 
111 
1111 

111 
1111 
1111 
111 

11111111 
111 

1111 
1111111 

inn 

mill 

mi 

mi 

mi 

11111 

in 

mi 

n 

li 

in 

in 

n 

i 

i 



Acad* Yr. 



1 
1 
1 
2 
3 
3 
4 



4 

7 
5 
6 
4 
4 
4 
5 
3 



K-15 



N-25 



A Priori Letter Grade 
Distributer 

A - 15 per cent 
B - 25 per cent 
c - 40 per cent 
D - 15 per cent 
F - 5 per cent 

Total 100 per 



Cut-offs for Letter Grades 
t frost frequency distribution) 

A, 580 and above 

B, 520 to 579 

C, 430 to 519 

D 380 to 429 
F 379 and bexov 



N-42 



4 

2 

2 D 
3 

3 N-14 



1 F 
1 H-4 
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Tests and HeasuTament Office 
Language Center 
Kuwait University 

M-25 



Staff Assignments in Connection with the Administration 
of Placement Tests and the Processing of Answer Sheets 

A. Roles associated with the administration of the tests. 

1. Supervisor of the Test Administration 

The Unit Supervisor usually assumes this job. It 
entails overall planning, supervision and coordination 
of the activity. 

2. Assistant Supervisor of the Test Administration 

The Unit Test Representative generally functions in 
this capacity. Duties include, but are not limited 
to: briefing participating teachers on their duties 
and responsibilities; distributing test materials to, 
and receiving them back fran, haad proctors ; observing the 
various testing rooms before and during the tests to 
assure satisfactory physical and administrative 
conditions. NOTE : If a new Unit Test Representative 
is to begin his or her duties in September and if the .UTR 
for the prior academic year is still on the Unit staff, 
the prior UTR should assume the aforementioned 
responsibilities, assisted by the new UTR* 

3. Head Proctors (Proctors-in-Charge) 

A Head Proctor is to be designated for each testing 
room. The head proctor is responsible for receiving 
from the UTR the test materials for his room, pre- 
serving their security during the test, and returning 
them to the UTR following the test. The head procto 
is also responsible for reading oral instructions, for 
timing the various parts of the test, and for general 
supervision of the proctors assigned to his or her 
room. 
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4 . Room Proctors 
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At least one room proctor (in addition to the head 
proctor) is to be assigned to each testing room for 
every 25 examinees, or fraction thereof. Proctors are 
to assist in distributing and collecting test materials, 
in preventing cribbing, and in assuring that students 
follow Instructions correctly. 

5 . Floating Proctor 

A floating proctor is an Arabic-speaking person whose 
major task is to answer questions that examinees may 
have who are unable to comprehend instructions in 
English. One floating proctor is to service two or 
more testing rooms, depending on the capacity of the 
rooms. In making head-proctor and room-proctor assign- 
ments, the supervisor should attempt to assign at leart 
one of his or her Arabic-speaking teachers to each test- 
ing room, thereby minimizing ti>e need for floating 
proctors. Units that will require the services of 
Arabic speakers from outside the unit are to notify 
Nr. Sala h Sabry as soon as possible of their requirements , 

B. Roles Associated uith the Processing of the Answer Sheets 

1. Supervisors of the Test Administration (see A. 1). 

2. Assistant supervisor of the Test Administration (see A. 2) 

3. scorers of the multiple-choice parts of tne test. 

4. Readers of the essay and/or supply-type parts of the 
test (if applicable) . 

5. Tabulator. 

The tabulator assembles the test-score data in a forn: 
that enables the T*M office to make a quick check on 
the reasonableness of the cut-off points that have 
been proposed on the basis of the norming administration 
and other relevant information. See M-26 of the T&M 
Office Test Development Manual for instructions concern- 
ing the use of the -Placement Test Tally Sheet." 
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Dftte : . Jon* *3 , J.980 _ : 

Rcf : WST _ _ . ^ jy 

Fm Victor W. Mason 

X 0 Director, Asst. Directors, Supervisors, Coordinators, UTRs 
****** M-26 of Test Development Manual : Placement Tggt Tally Sheet 



1. Enclosed is a copy of the Placement Test Tally Sheet 
for the use of tabulators (generally the UTRs) in assembl- 
ing the score data from the administration of the placement 
test in a form convenient for checking on the reasonableness 
of previously suggested cut-off points. These cut-off 
points are determined primarily but not exclusively 

on the basis of the results of the norming-test administra- 
tion involving the same test items and sections taken the 
previous academic year by students enrolled in the same 
courses into which newly entering students are to be 
considered for placement. 

2. The tally-sheet form consists of six pages for the 
entering of test-score data. It has been prepared to 
handle placement tests which go up to a maximum of 160 
points* If any future placement test forms are longer in 
terms of time or actual length , or both, so as to exceed 
160 points, one or more pages would have to then be added 
to accommodate the greater length up to the new point 
total. 
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3. The tabulator begin* his task by turaina to the 
last page (page 6) of the form, on which the results of 
students with the lowest: possible placement test scores 
are registered with tally marks in column 2* If no 
student has a placement test score as low as 10, the 
tabulator turns to page 5* If , as a hypothetical case, 
the lowest mark on the test is 23, the tabulator places 
a mark in column 2* If only one student obtained this 
score, the numeral "1" is written into column 4 

( "FREQ* , " for "frequency*) . This number (1) is then 
added to the total of all scores below it (i.e., 0) to 
give a cumulative frequency of 1, which is written into 
the corresponding blank in column 5 ("CUM, F. m ) • If 
some 280 students took the placement test, the percentage 
of all students with this score (23) and below is then 
calculated (1/280 * 0*34%) and written into the proper 
blank in column 6("CUM. %") . 

4. If the next lowest score is 29, and if 3 students 
obtained this score, 3 tally marks would be placed in 
oolum 2 beside this score* The numeral "3* would then 
be written into the appronriate blank in column 4 . The 
total of 3 would then be added to the previous total of 
all scores lower than 29 (i.e., only I) to give a new 
sub-total of 4 and written into the blank in column 5* 
The cumulative percentage of th*s level would then be 
calculated (4/280 - 1*4%) and entered into the appropriate 
blank in column 6* 

5. This procedure would then be followed until all 280 
students had been accounted for, or 100 per cent of the 
group taking the placement test. The supervisor and UTR 
are then prepared to examine the previously recommended 
cut-off points for the entering class, as suggested by the 
previous year's norming test results and subsequent .report* 
Determination of thaes cut-off points is then made in the 
light of a priori administrative decisions of previous 
years as to roughly what percentages of students are expected 
to be placed into each of the various required courses of 
the unit* 
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RANGE: 160 - 131 
TABULATOR 

TEST TEST 



SCORE TA T . LTK S SCORE FREQ. CUM. P. CUM.% 



160 




160 








159 




159 








15o 




158 








157 




157 








156 




156 








155 




155 








154 




154 








153 




153 








152 




152 
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150 




150 








149 




149 








i48 




148 








147 




147 








146 




146 








145 





145 








144 




144 








143 




143 








142 




142 
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140 




140 








139 




139 








138 




138 








137 




137 








136 












125 




135 








134 




134 








133 




133 








132 




1?? 








131 




131 ' 




* 





TALLY SHEET 

FACULTY 
TEST FORM 

DATE 



hi 




FACULTY 

RANGES 130 - 101 TALLY SHEET ^ 

iWA - TEST TO! 

TABULATOR 

DATE 

TEST TEST 



SCORE TALLIES SCORE FREQ, CUM.F. CUH.% 



139 


r ' ■ ' 'i ^= 1 • • • • - ■ 1 


120 


~ 






12? 




129 








12S 




12C 








Li. I 




127 








126 




126 






i 
1 


US 




12S 






i 


124 




124 






1 


123 




121 






i 


1 

122 




122 


i 






121 




121 








120 


1 

12Q 


1 ! ! 


119 




119 ! 






l 

! 118 




118 




1 


; 1« 




117 




1 

1 J 


i lie 




\ 1 c 






i 


: 115 




115 








! 114 ! 


114 








113 ! 


113 








112 
1— 1 




112 








111 




.^li- 
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KUWAIT UNIVERSITY Wml 
lamcuacb asm ri an fy 

T 1 IT' M-27 u-UBii C*JL~i>* 

Due : April l,_12fil : 

H C ( . TIM Memorandum of November 15, -1980 : ^ j\ 

MEMO 

Prom Victor W, Mason ph^/ 

X 0 Holders of T&M Manual 

Subject Pre-Test Cover Page: M-27 



This memo is a follow-up to that of November 15 , 1980, 
concerning a proposed format for a suitable cover page for 
copies of new examination forms to be pre-tested within a 
unit as part of the normal review process for those exams. 
The memo of November 15 , 1980, solicited suggestions for 
further modifications that staff members might regard as 
useful or necessary. 

The enclosed form may now be considered the final 
form of the cover page for pre-test copies. Please insert 
the page into your manual as M-27. The entry in ycur 
Table of Contents may be entitled *Pre-Test Cover Page." 
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PT.KASE HANDLE CAREFULLY 



P RE-TEST COVER PAGE 



A. Identification Data 
I. Nana of Course t (e.g.) Science 162 



2. Nam of Examination: (e.g.) Final Examination 

3. Nam of Testee: ; 

4. Date Distributed: 



5. Date to be Returned 

to Teat Development Committee: 

6. Number of Teat Booklet: 



7. Pretaat Form: Q Preliminary (First) 
r*| Revised (Second) 
□ Other (e.g.. Third) 

B. Security Instructions 

1. Please handle this pre-test form very carefully. Do not 
leave it lying around your office exposed where students 
may see it. 

2. Please return it to the appropriate Test Development 
Committee in the same envelope in which it was distributed 

to you. 

3. It is best to receive this form by hand from the appropriate 
person concerned and also to hand it back to the saiL* person; 
to avoid later wrangles over tests which were allegedly left 
somewhere for someone but subsequently disappeared. 

C. Procedural Steps 

1. Take this test uncritically just as a student would take 
it, answering all questions as instructed. 

2. Put a check mark or question mark beside any line in which 
you find an obvious error, a source of confusion or some 
other problem indicating possible weakness in the test, 

in your opinion* 

3. Finish taking the test like a student before addressing 
possible problems. 

4. When you have finished taking the test, turn to the last 
page of this booklet, which is blank. On this page, please 
elaborate on any points in the test-form of which you think 
you would recommend modification to the Teat Development 
Committee. 



PLEASE HANDLE CAREFULLY , 
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KUWAIT UNIVERSITY 
LANGUAGE GENRE 




OAtt 



M-28 



Date: 

Ref : 

From 
To 



April 1, 1981 _ 

Victor W. Maaon 
Holders of TftM Manual 



MEMO 



Subjcci Addition of M-28 to T&M Manual , M-28; 
Purpose of the Worming Test at KULC 



At the suggestion of Dr. Malcolm, the enclosed 
memorandum entitled "Purpose of the Norming Test at KULC 
is being inserted into the T&M Manual as M-28. Please 
add this title also to the Table of Contents of your copy 
of the manual. Thank you. 
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Purpose of the Nonning Test at KULC 



1. Kuwait University Language Center requires all students 
newly enrolled in a college to take the English Placement 
Test (EPT) of the English Language Unit (ELU) of that 
college. The main purpose of the EPT is to attain maximum 
homogeneity in English proficiency of students at each 
course level in the sequence of required English courses 

of each ELU. 

2. The EPT in each ELU is administered twice each academic 
year? i.e., at the start of each semester. Each such 
administration of the EPT is known as the "operational 
administration" of the test? that given in September is, 

of course, the more important, since the great majority 
of new students enroll in a college in the fall following 
their graduation from high school the previous spring. 

3. In order for the Language Center to determine the 
suitability of its Placement Tests prior to their operation- 
al administration, a full-scale pretest of each EPT is done 
during the academic year preceding the scheduled date of 
operational administration. For example, EPTs given in 
September 1980 were pretested in December 1979. The 
subjects of the pretest in all units were continuing students 

i.e., those already enrolled in the same required English 

courses that new students would be placed in on the basis of 
their EPT scores the following academic year. 

4. This pretest is known as the Norming Test (NT). The 
results of "norming" the test provide an indispensable 
standard by which to compare the English proficiency of 
incoming students and continuing students. Thus, the NT 
is extremely important to the staff members of each ELU 

in providing them a means to achieve a reasonable degree 
of homogeneity in their EFL classes. 
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5. As all experienced foreign- language teachers know, 
the first requirement of a rational program is sufficient 
similarity of ability levels in a class to make effective 
instruction possible for all students. In poorly organized 
classes f a significant number of students will be wasting 

their time r and will therefore become frustrated either 

those who are much better than the majority and will thus 
quickly feel bored? or those who are much weaker than the 
majority and will therefore eventually give up; or both. 
The likelihood of teacher frustration in programs with such 
classes is also very high. 



6. KULC Norming Tests are crucial in the effort to attempt 
to minimize such frustration on the part of both students 
and teachers. They are crucial , therefore, in our efforts 
to promote a foreign -language program of professional quality. 
They help to do so in three ways: 



a. by providing an empirical basis of comparing 
the English proficiency of incoming students 
with that of continuing students 

b. by acting as a common test for all continuing 

s tudents already enrolled in an ELU's sequence 
of required courses 



c * by permitting statistical analysis of the items 
and the parts of each test prior to its opera- 
tional administration. 
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Te3ts and Measurement Office 
June 1980 



A. History of the Office 

1. The T&M Office of Kuwait University Language Center 
was founded by Dr. Donald J. Malcolm in the fall semester 
of the 1977-78 academic year. Dr. Malcolm was the head of 
the office during the three full years which he spent with 
the Language Center, from September 1977 until June 1980. 

2. During those three years, he was assisted in the 
office by several Language Center staff members: Mr. Steven 
Spurling, Miss Sana 'a Al-Naqeeb and Mr. Victor Mason. 

Mr. Spurling assisted full-time in the spring semesters 
of 1978 and 1979. Miss Naqeeb was with the office during 
the spring semester of 1979. And Mr. Mason assisted full- 
time during the 1979-80 academic year. 

3. The T4M Office has worked closely, in the development 
of tests, with all of the English- language units of the 
Language Center as well as with the French Language Unit. 
Nearly all test-development work until now has concerned 
preparation either of placement examinations by the five 
ELUs nd by the FLU or of final examinations of required 
courses in the ELUs. 
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B. Liaison with the Units 



4. Liaison between the language units and the TtH 
Office has been the responsibility of the unit test 
representatives (UTRs) , languaqe teachers in the units 
who are appointed by their supervisors to coordinate 
testing activities both within their units and between 
the T*M Office and their respective units. The consider- 
able amount of time that the job entails has required 
substantially reduced teaching loads for the UTRs. 

C. Preparation of Placement Tests 

5. The UTR has the primary responsibility in the 
unit for over-seeing and coordinating development of 
new forms of the unit's placement examination for the 
fall semester of the year, when the majority of new 
students enter the university. Preparation of the new 
placement examination begins one full calendar year 

in advance of its administration , according to the follow- 
ing schedule: 

a. early September t preparation of -Test Develop- 

ment Schedule" by T*M Office 

b. September-November ; Preparation of test 

1) writing of items and sections in unit 

2) review of items and sections in unit 

3) review of items and sections in TtM Office 

4) final revisions in unit 

5) preparation of two parallel one-hour 
half-tsst forms , the "norminq test" 

6) completion of tefft-item assembly form (TIAF) 



c. late November : printina of normana test in 

AVA Office 

d. December: administration of norminq test 

c. December- J anuarv : computer analysis of 

norminq test results 
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f. February; analysis of norming test results, 

and recording of statistical character- 
istics of items on item cards kept 
on file in T&M Office? estimating 
cut-off points on fall placement test 

g. March ; correction or replacement of faulty 

items and sections, as determined by 
statistical analysis; review of corrections 
by Tests and Measurement Office 

h - late Mar ch-early April : assembly of the two 
one-hour tests as the two-hour fall 
placement test; preparation of NT report 

i. late April; printing of fall placement test in 
AVA Office 

j. May: packaging, labeling and storing of 
placement tests, scoring keys and 
instructions for administration under 
secure conditions in the unit 



k. Ma^: spring administration of norming test, 
if necessary (as in ELU Engineering) , 
and combination of results with those 
from earlier administration fcr establish- 
ment of cut-off points on upcoming fall 
placement test 



1* September ; administration of placement test and 
completion of tally sheet of results 
to aid in determining cut-off points 
between classes 



Following administration of the Dlacement test, 
the results are computer-analy2ed and the TSM Office 
prepares a report on the results of the test. The UTR 
records the statistical characteristics of each item 
for that administration on the item cards. 
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D. Preparation of Final Examinations 



7. The second major area of test -development activity 
by the UTR concerns preparation of his unit's final 
examinations for required courses. The UTR is an ex 
officio member of all test development committees (TDCs) 
working on those finals, and keeps the T&M Office informed 
of the progress of the development of those finals. 

8. Development of final examinations for required 
courses in each semester involves the following procedures: 

a. preparation by the T&M Office of a "Test Develop- 
ment Schedule" for each unit's final exams 

b. submission of these schedules by the UTR to 
his supervisor and coordinators for their 
concurrence 

c. appointment by each relevant coordinator of a 
TDC from among instructors teaching the course 
that semester, with the coordinator as chairman 

d. revision of the statement of instructional goals 
(behavioural objectives) of the course in the 
event of curricular changes during the given 
semester 

e. preparation of the Final Exam Content Specifica- 
tions (PECSs) , the "blue-print" of the language 
skills to be assessed, the kinds of test items 
each section will contain, and the relative 
weight to be assigned to each 

f. discussion between the T&M Office and the UTR 
of 1) the behavioural objectives and 2) the 
FECSs for the new finals in the light of a 
review of past finals for the same courses, 
with either T&M approval or recommendations 
for changes 
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g. following agreement on the FECSs between the 
T*M Office and the TD Committee , writinq and 
reviewing of items for the new final exam, with 
the technical assistance of the UTR 

h. following assembly of the new test, review of 
the preliminary finished form by all instructors 
teaching the respective course and by at least 
two others not teaching it 

i. following all review procedures, submission of 
the final exam to t'.e unit supervisor for 
preliminary approval 

j. submission, together, of all final exams of a 
given unit to the T&M Office for review 

k. modification of the final by the TO Committee 
following recommendations by the T&M Office 
mutually agreed upon 

1. submission of final by TD Committee to unit 
supervisor for final approval 

m. submission of final by unit supervisor to T&M 
Office for final review of test format, answer 
sheet design, and scoring procedures* 

9. The various units have generally arranged to have 
printing of their final examinations done without the 
assistance of the T&M Office. Moreover, the statistical 
analysis of final exams has, thus far, seldom been under- 
taken, with the notable exception of the final examination 
for the students of the 093 (intensive) program of the 
Engineering ELU. 

10. It has been arranged, however, that this summer for 

the first time computer analysis of the Language Center's 
final exams will be performed at the university's Computer 
Center on the Adeliyah campus. Once installation of the 
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new and enlarged Computer Center has been completed on 
the Khaldiyah campus at the end of the 1980 r*\m~*~r yaarf 
analysis of the Language Center's final examinations 
at the Computer Center is expected to become a routine 
procedure • 



E * Assistan ce of the Cownuter Cmi-mr 

.1. Up until calendar year 1980, the TiM Office has 
obtained the indispensable assistance of the Computer 
Center in standard item and test analysis. This has 
been of value primarily in two respects: 1) evaluating 
achievement of language skills by students, and 2) 
2) evaluating the effectiveness of tests themselves in 
performing that task. 



.2. The new Computer Center on the Khaldiyah campus 
will be equipped with an optical scanner which will -read- 
answer sheets up to a maximum capacity of 25,000 sheets 
per hour. This device will relieve the Language Center's 
teachers entirely of the need to score objective tests 
such as the units' placement tests, as well as large 
parts of the finals of many courses* 



.3. Answer sheets will become standardized, to utilize 
only a limited number of formats. Early in the fall 
semester of the 1980-81 academic year, the T»M Office is 
to consult closely with the personnel of the Computer 
Center to work out the specific formats for answer sheet 
design desired by the ELOs of the Lanugage Center. Once 
these formats have been settled upon, printing orders will 
be sent from the Computer Center to the printers in 
England. 



.4. As part of the expansion of Computer Center services* 
terminals will be installed in many offices of Kuwait 
University. One such terminal has been requested by the 
Language Center to facilitate the storage and convenient 
retrieval of data on students relevant to Language Center 
needs. A terminal will greatly aid the Language Center 
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in several ways. For example, at registration time an 
official can see at a glance all language courses taken 
by a given student. The student is thus much less likely 
to register improperly for a Language Center course , such 
as to attempt to bypass prerequisite courses, as has 
happened not Infrequently in the past. 

15. The terminal will be invaluable also for permitting 
the Language Center to have ready access to each student's 
entire record of course marks and his scores on aljL major 
-Language Center-level" tests, such as placement tests, 
mid-term tests, "unit" or common tests, norming tests and 
final examinations. The convenience and usefulness of 
this new system for record-keeping, for data retrieval 
and for evaluation purposes can hardly be overestimated, 

F. The Test Development Manual 

16. During the years 1977-80, Dr. Malcolm prepared a 
large number of memoranda to guide the units in the 
development of professional-quality language tests. 
These many memos have been assembled to constitute the 
T*M Office's Test Development Manual . 

17. Some of these memoranda deal with the duties of the 
UTRs, coordinators and supervisors in the development of 
the placement tests and final examinations. Others deal 
explicitly with the procedures to be followed in the 
writing and reviewing of specific types of test items 
and sections* The guidelines deal with both objective- 
type and supply-type testing formats. 

18* The manual provides guidance in the specification 
of curricular goals of the Language Center's courses. 
It provides assistance to the staff in interpret- tm the 
results of computer analysis of major examinations. It 
gives a detailed explanation of how to fill out forms for 
the processing and summarizing of essential test data. 
It explains how to compute standard scores, to minimize 
the effect of teacher subjectivity in the asslanlng of 
grades to students enrolled in courses with multiple 
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sections whose teachers may vary widely in their grading 
practices. 

G. Record-Keening systems 

19. The T*M Office has several types of record-keeping 
systems. The most important ones are the filing system, 
the card catalog and the computer print-out records. 

20. The files maintain a complete set of old ELU test 
forms for the past three years, both for placement tests 
and for the final examinations of required courses. 
Included with the finished test forms (both base and 
scrambled forms) are answer sheets, answer keys, scoring 
transparencies and rationales for the scoring of supply- 
type material. 

21. The files for placement tests also contain reports 
of the results of those administrations. Separate files 
contain norming test forms i reports on the results of 
those test administrations, as well. 

22. The files for course-final examinations are accompanied 
by additional files stating curricular objectives and the 
Final Exam Content Specifications for each course. These 
are kept updated each semester. 

23. A large part of the filing system is given over to 
Language Center, unit and TIM Office administration. Several 
sections of files are primarily dwoted to research and 
evaluation reports prepared by Dr. Malcolm, some of these 
reports concern programs outside the Language Center, 

such as studies done for the College of Graduate Studies 
on the basis of experimental administrations of the TOEFL. 
Much space in the filing system is unfortunately taken up 
by enormous numbers of answer sheets. 

24. The card catalog is maintained to provide an item 
bank for the preparation of new placement test forms. All 
placement tests of the Language Center are in a multiple- 
choice format, and the items— primarily to test grammar, 
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vocabulary and reading comprehension— are maintained 
each on a separate card. The card catalog system of 
placement test items was meant to serve as a model for 
the development and storage of final examination items 
as well, but until now this ambition has been unfulfilled. 

25. All computer analysis of Language Center test 
results has been performed on either placement tests or 
norming tests. The Engineering 098 final multiple-choice 
test used to be analyzed only because it was the unit's 
norming test for 098 students as well. These results 

are maintained on computer printouts bound into book form. 
The need for greater efficiency in binding these printouts 
will increase now that many course -final examinations are 
being computer-analyzed for the first time during the 
summer of 1980. 

H. Continuing Problems 

26. Probably the principal difficulty for the T&M Office 
semester after semester is a too-limited understanding by 
many members of the Language Center's teaching staff of 
what the office is attempting to do. Characteristic of 
the problem are statements from course coordinators that 
suggestions from the T&M Office result in a lot of extra 

work for the units the implication being that much of 

that work may be unnecessary. These complaints generally 
increase toward the end of a semester when pressures on 

TD committees to finish their final exams on schedule are 
greatest. 

27. The subject of development of final examinations 
for required courses is without doubt the source of more 
frustration to the TftM Office than any other. The reasons 
for dissatisfaction are several, all interrelated. First 
and foremost is the problem that many TD committees simply 
do not adhere to the test-developement schedules prepared 
at the start of a semester by *rhe T4M Office. 
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28. This failure to adhere to the test-development 
schedules not infrequently results in final exams being 
submitted late to the T&M Office for its review, when 
recommendations for chanqe can no longer be implemented. 
Experience has demonstrated reneatedly that the more tardy 
the completion of a final exam, the more likely that test 
will suffer from major structural flaws. Such weaknesses 
are the consequence of neglect of probably the most 
important part of the test-development process: a 
sufficiently thorough and adequate review, involving pre- 
testing, thorough criticism, rewriting and further review 
and revision. 

29. One of the principal reasons for weak tests is the 
very uneven abilities of E "L teachers in preparing accept- 
able tests: many are very good at writing test items and 
sections but many others, through lack of training or 
cxoerience, or both, are less so. The recommended system 
of Dreparing final examinations by committee over several 
months is intended to try to neutralize any test-writing 
deficiencies of Language Center staff. Gr^elines to the 
writing and reviewing of tests are laid out explicitly in 
memos of the TD Manual written by Dr. Malcolm to assist 
the staff. From the anpearance of many final examinations, 
however, it appears as if those guidelines are too in- 
frequently consulted. 

30. It is widely known that the preparation of good 
multiple choice items is difficult and time-consuming. 
Language. Center finals resorting to this format, especially 
to test reading comprehension and grammar, often are 
insufficiently reviewed to remove or improve obviously 
weak items. 

31. Many final examinations in the past have not been 
comprehensive enough. it has been possible in some courses 
for all students, even the weakest, regularly to complete 
their finals in less than two hours, for courses that 
have met every day of the semester. 

32. It is not difficult each semester to find a final 
examination for a higher course which is obviously less 
challenging than th« final examination for a low«r course 
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in the same unit. This Ls despite the fact that the lower 
course is supposed to be a prerequisite for the higher* 

33. Some TD Committees use the same test sections in 
the final examinations of their courses two or more 
semesters in succession. For testing reading comprehension, 
listening comprehension and writing, this practice can 
clearly compromise test security. Such frequently used 
sections will be discussed by students and will give an 
unfair advantage to students in following semesters know- 
ing what to expect. 

34. In the Language Center, the T&M Office has the 
ultimate responsibility for assuring quality control of 
major "Language Center- level" examinations. The office 
cannot f however , take responsibility for final examinations 
that it receives too late to review, prior to test 
administration . 

35. The FE development process begins at the start of 
a new semester, so it does not seem unreasonable to ask 
that the preliminary finished form be submitted within 
2\ to 3 months, as specified by the TD schedule and 
agreed to by the TD Committee. 

36. The limited resources of the T&M Office necessitate 
a schedule for review and revision of FEs that is early 
and long enough to make the orderly review of the final 
exams for all required courses possible. This process 
should allow a minimum of two weeks when the T4M Office 
can give its undivided attention, during office hours, to 
the completion of all reviews of finals, the preparation 
of summary reports (when necessary or desirable) and 
incidental discussions with concerned UTRs and/or TD 
Committees • 
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A Brief Biography of 
Dr. Donald J - Malcolm 



Dr. Donald J. Malcolm received his M.A. irom the 
University of Rochester in 1947 and then taught courses 
there for a year in statistics and psychology for teachers. 
After that, he attended Harvard University , where he was 
a teaching fellow and research associate at the Center 
for Field Studies of the Harvard School of Education, 
obtaining his Ed.D . in 1951. 

Following graduation, Dr. Malcolm worked in program 
evaluation for the Human Resources Research Institute of 
the U.S. Air Force at Maxwell Field, Alabama, from 1953 
to 1959. He joined the Educational Testing Service (ETS) # 
at Princeton, N.J., in I960 and remained there nearly two 
decades • 

While with ETS, Dr. Malcolm held a series of highly 
responsible positions. He became assistant director for 
test development, responsible for (a) the College Board 
Achievement Test and (b) the Advanced Placement Test. 
From 1967 to 1969, he was advisor to the Board of Inter- 
mediate and Secondary Education in Bangladesh, where he 
was Program Officer for the Ford Foundation. 

In 1969, Dr. Ma Lcolm took over as the third director 
of the world-wide TOEFL program, following David Harris 
and Leslie Palmer. lie held that position until 1977, 
longer than any other TOEFL director thus far. 

From 1977 till 1980 Dr. Malcolm was head of the 
Tests and Measurement Office of Kuwait University Lanquage 
Center. He left in June 1980 and it is hoDed he will 
return in the future as a consultant to the Language Center. 



VM/st by Victor W. Mason nff4ce 

Tests a Measurement Office 
Lanauaae Center 
Kuwait University 
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KUWAIT UNIVERSITY (f&fel *3*jfll 

LANGUAGE CBVI*E L^^Jfcf *J±m 

Tote mmi Mwt OOct M-30 ^UiH^ o)jl_r*>?1 4-3* 

tfctc: May 26, _ _ . : 

Rcf : Memorandum of 11 May 1980 to Unit Supervisors ... : ^ J\ 

MEMO 

fwm victor w- Mason ' \f2' 

T« All Holders of TP Manual 

Subject Security Systems fo r 1) Teats Baaed upon the Multiple-Choice 
Format and 2) Items Banks 



The enclosed new memorandum to be inserted into the 
Test Development Manual as M-30 is concerned with the fol- 
lowing policies, practices and procedures for maintaining 
and assuring security for KULC examinations heavily dependent 
on the use of the Multiple-Choice format: 

A. security for item banks 

1. the importance of student ignorance about the 
existence of item banks 

2. the numbering of test booklets and answer sheets 

3. student requests to review their tests 

4. the destruction of old test booklets 

5. storage systems for item cards 

B - security procedures for the administration of 
major examinations 

1. the use of alternate test forms 

2. the use of teat-designation and form-designation codes 

3. test-distribution patterns to foil student cheating 




4. Mating of ttudtntv by teachers , and 



5. realistic timing of test parts, whether formally 
or informally. 



2. It cannot be emphasised too sti/omgly that the fcangueg& 
Center's first line of defease acalra'* widespread cheating 
on exams and compromise of test security systems remains student 
ignorance as to what those systems are. Given the emphasis 
on tests and grades at Kuwait University , it would be naive to 
assume that test results and grading considerations, unfortunately, 
are not the primary concern of a very large number of students 
enrolled in university courses that are within the Language 
Center's jurisdiction. Tor those determined to use underhanded 
means of trying either simply to pass a course or to obtain a 
grade they do not deserve , knowledge concerning the heart of 
our security systems may be valuable indeed. 

3. It is my strong request, therefore, that to prevent large- 
scale cheating in an examination room henceforth, KULC staff 
be extremely reticent with students on any subject concerning 
security systems for maintaining the Integrity of future exam- 
inations depending extensively on the MC format. Staff should 
never discuss these subjects with the students, should avoid 
telling them which teachers of a course are most responsible 
for development of course common exams (e.g., members of the 
Test Development Committees) , and should avoid telling them 
anything about any office or location where tests may be 
developed, stored, or reproduced; i.e., as regards their own 
units, the AVA offices and the Tests and Measurement Office. 

4. The students would in any case be curious about these 
subjects. Teachers should write down the names of students 
going out of their way to ask them improper questions relating 
directly to test security systems and procedures, and should 
give those names to their unit supervisors. It goes without 
saying that staff members raising such subjects with students 
would not be acting xfi a responsible and professional manner. 

5. It is hoped that these matters can be clarified for all 
staff members of the unita. It is also highly desirable that 
awareness of KULC policies and practices of test security hence- 
forth be made an integral part of the orientation procedures 
for new KULC staff members. 
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To Holders of TP Manual 

Subject Security Systems for Tests Based upon the Multiple-Choice 

Format and for Item Cards 



A. Introduction 



1. The multiple-choice (MC) testing format is one of the 

most useful developed in recent decades but also one whose 
security is among the easiest to compromise. The fact that 
MC items are also among the most difficult, time-consuming 
and expensive to prepare makes close attention to security 
systems for protecting MC tests and item banks essential. 



2. The extra efforts required to protect the integrity of 
the system are repaid many times over by the time saved for 
teachers in subsequent test development work and by the 
demonstrable improvement in test quality over time. The 
data recorded on the basis of test and item analysis are also 
one indispensable basis for undertaking meaningful program 
evaluation and revision. 



3. In the Language Center , an item bank new exists for all 

Norming Tests and Placement Tests used by the Mils. Therefore , 
stringent security procedures are followed before, during and 
after each such administration. Now as iter? banks are developed 
for the final examinations for many KULC required courses 
(and even for some major progress tests: mid-term exans, "unit 
exams," etc.), it will be necessary for similar procedures 
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to be instituted by the Test Development Committees (TDCs) 
for these courses. 

B. Security for Item Banks 

4. it is, therefore , not too soon to specify the indispensable 
security precautions that must be taken f to avoid many hundreds 
of man-hours of teachers 1 test-development time going for naught. 
It should be apparent , for example , that the disappearance of a 
single test, booklet , before or after an exam, can wipe out a 
large part of a TDC's item bank . Given the determination of 
many of our weaker students to pass their finals , even if by 
dubious means , KULC staff members must do their utmost to safe- 
guard test security at all times. 

1 . Student Ignorance about Item Banks 

5. The first security requirement for safeguarding a course 
item bank is student ignorance about its existence. A test 
booklet that disappears following administration of an exam 
obviously becomes much more valuable to a student and to his 
friends if they can be sure that the identical items will 
reappear in the future. Just one teacher thoughtlessly 
discussing the existence of such a system with students compro- 
mises the security of tests not only for his or her course; 
students would be quick to assume that a similar system exists 
also for other courses. Students transferring from one college 
to another could quickly transmit this useful information and 
put a premium on the assembly of KULC test files, much as test 
files are accumulated at universities abroad (e.g., by frater- 
nities in the United States). Thus, discussion about the system 
by a teacher with his or her students would be very irres- 
ponsible and unprofessional. 

2. Numbering of Test Booklets and Answer Sheets 

6. One essential procedure for guaranteeing security for an 
item bank bexore and after test administration is to number 
all test booklets and attached answer sheets. This not only 
greatly simplifies the packaging of new test forms and their 
distribution and collection in the examination room; it makes 
it possible to trace any missing booklet and/or 
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answer sheet, especially in double-checking tests when handed in, 
in the examination room, and in triple-checking them later. 

7. Teachers responsible for administering a test can more 
easily check for accuracy before the test administration begins, 
to see that the totals and numbers written on the outside of the 
test package correspond to its contents. Inside the testing 
room, they can distribute their tests in serial order and collect 
them later in a similar manner. Furthermore, the numbers on the 
test booklets and answer sheets greatly facilitate the efficient 
double-checking of totals at the end of the exam, when students, 
having been instructed to remain in their seats, begin talking 
and when confusion may thus set in. These procedures are already 
followed in the case of all KULC Placement Tests and Norming Tests. 

8. The fact that both the test booklets and the answer sheets 
are numbered has several distinct advantages. Students who 
absent-mindedly (i.e., innocently) carry off a test booklet 

(it does happen!) are highly unlikely to forget to turn in their 
answer sheets; the latter have their names on them so they may 
be rapidly identified and the booklets recovered. For those 

rarer cases of students aware they are failing badly who 

might try to conoaal both booklet and answer sheet in their 
clothing (to prepare for a later day), numbering is not enough; 
a student attendance roll which all students sign (ideally, in 
the presence of their own teachers) is also necessary. Simply 
counting the total number of students is not enough, especially 
when weak students try sometimes to have friends take their 
tests for them. 

9. One necessary procedure for reducing the likelihood of loss 
of a test booklet after test administration is to tell students 
before the test that they may take nothing to their seats except 
writing implements. Books, purses and other such personal 
belongings should never be permitted to be carried to their 
chairs . 
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3. Students' Reuueaua to Review Their Tests 



10. A major security risk to any test system occurs when 
students are permitted to review their tests following admini- 
stration and correction. The risk is two-fold: one, the 
physical handling of the booklet, which may disappear; and 
two, the opportunity for students to study closely the contents 
of the booklet, even when they hand back the test form to the 
teacher. 

11. At KOLC, this risk is less in the case of final exam- 
inations 9 since these are never returned to the students and 
may be seen only by special request in the office, and under 
the supervision, of their teachers. The risk is far 
greater in the case of course non- final common progress tests 
(midterms, H unit tests," etc.), since it is a reasonable part 
of teaching procedure for these to be brought into the class- 
room and distributed for the students 1 inspection Test 
contents may then be fixed much more firmly in mind, especially 
the vocabulary items, reading passages and listening passages. 

14. Moreover, if an item bank is to be prepared for such 
non-final common progress tests, the teachers must again collect 
all of the test booklets and answer sheets and count them 

both in class before the students leave the classroom and 
later in their offices. This task is complicated by the fact 
that, when students from different sections are mixed randomly 
for the specially scheduled administration of common exams, 
students of the same class will not have test booklets numbered 
in sequence. It is also much more difficult during ordinary 
classes to prevent students' personal belongings from becoming 
mixed up with the tests themselves, particularly amidst the 
commotion reigning at the end of a typical class period. 

15. For these reasons, item banks for course final examinations 
must bo kept separate from those for all other course common 
tests* They cannot be mixed. Secondly, the ease with which 
students remember the contents and results of pre-final common 
exams means that the Items and sections of these tests can be 
safely re-used far less often than can those of final exams, in 
the case of a final exam, use of a vocabulary item or a reading or listening 
passage (or writing task) should probably not be repeated more than once 
every throe or four semesters (i.e. , about once every two years) • In the 
case of pre-rinol ex mini exams, a safe fftanuard would probably be to extend 
these Urns restrictions by 50 to 100 per cent (i.e., once every three or 
four years 1 * 
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4 . Destruction of Old Teat Booklets 



16. Since every old test booklet developed on the basis of 
an item bank becomes a potential security risk, immediate 
destruction of tests becomes imperative. It is inconceivable 
that they would simply be left lying about for days, weeks 
and even months at a time. All old test booklets except 
for those needed as file copies for record-keeping or needed 
for special make-up test administrations should be destroyed. 

17. In the 1981-82 academic year, the KULC AVA Office will 
have a powerful new shredder installed for this purpose. 
This machine will be an exact duplicate of that now in the 
T*M Office, which is available for the destruction of all 
unit placement tests and norming tests. 



5. Storage of Item Cards 

i. Every ELU will need a special cabinet for storing its 

item cards, similar to the one in the T&M Office which holds 
KULC Placement Test items. Coordinators with temporary 
security problems for their cards may keep them at home until 
their unit's own cabinet has been acquired. 



c - Security Procedures during the Administration 

of Tests 

I. Security policy and procedures in effect during test 

administration rely most for their success upon the same 
basic principle emphasized in the above discussion on item 
banks: student ignorance of their existence. The procedures 
and practices employed in the case of major KULC exam- 
inations depending heavily upon the multiple-choice format 
are primarily the following three: 

a. the use of alternate test forms, such that multiple- 
choice items of the base ("A") form are scrambled 
in the alternate form(s) 



b. the use of test designation codes familiar to the 

teachers but not the students distinguishing 

between the forms, and 
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c. distribution of test booklets such that no student 
sits adjacent to another with the same test form. 



In addition, the T&M Office recommends: 



d. seating of students such that friends may not 
have the opportunity to assist each other, even 
clandestinely , during an exam, and 

e. realistic timing of test parts, so that students 
prepared for the exam should be able to complete all 
all parts and sections. 



20. Security during the administration of MC tests is greatly 
enhanced by having alternate forms of the same examination? 
i.e., a base form and one or more "scrambled" versions (dubbed 
"A," "B," "C," etc.). Students cannot cheat from their 
friends 1) if there are different test forms being used and 

2) if they do not know who has the same test form as they do. 
Students should never be told about the nature of the 
differences in their tests or about the way in which those 
differences may be identified. 

21. This is one of the principal reasons behind the use of 
test designation codes at the top of the cover page of a test 
booklet. The code including the test-form designation (A, Q, 
X, etc.), is intended to be meaningless to the students. 
(The existence of such a security code will be pointless if 
the students are to have the code conveniently broken for 
them by being told how it works.) 

22. Therefore, for course common examinations , including final 
examinations, these letters are best not included with test 
page numbers throughout the test booklet. Page numbering that 
combines the form code and page number has been used in the 
past for convenience during the reproduction and collating 

of test booklets; it has no security value. In the future, 
it would be best that entirely different symbols (preferably 

unfamiliar to the students e.g., letters of a foreign 

alphabet) be placed in an inconspicuous page corner to 
distinguish between test forms. The same precautions should 
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be taken with future administrations of the Norming Test? 
and even with the Placement Test, since some incoming students 
seem aware of our alternate- form system for MC tests, and 
therefore have been known to object if not allowed to sit in 
a seat of their choice.. 

2. Distribution of Test Booklets 

23. The pattern of distribution of test booklets for major 
KULC examinations employing the alternate test-form system has 
followed the "diamond" configuration. That is, no student 
ever sits adjacent to another student with the same form. 
This arrangement should prevent pairs of neighboring students 
from assisting each other on these tests, since item numbers 
for the same tasks do not correspond. 

24. However, it sometimes happens that students learn of these 
arrangements for test-booklet distribution and for test codes. 
When they do, there is then the serious danger of mass com- 
promise of test security on the multiple-choice parts of an exam. 
This security breach comes about when as sometimes happens 

on KULC final exams a large group of friends aware of this 

system arrange to seat themselves such that a student good 
in English sits in the front of the testing room while his or 
her weaker friends sit in back in chairs with the same test 
form. They then work together in tandem through entire test 
sections or parts, with the one(s) in front giving surrepti- 
tious pre-arranged signals to the watchful host in back. 

25. Such large-scale cheating is easiest to achieve in MC 
sections with discrete-point tasks, especially grammar and 
vocabulary. More integrative tasks like reading are more 
difficult to cheat on because students work at much different 
rates and might not undertake reading passages in the same 
order. "Inherently timed" tasks like listening comprehension 
and dictation almost by definition prevent such cheating 
unless students flagrantly talk or whisper to each other. 

26. In the event of students attempting to circumvent the 
aforementioned, .security systems for MC tests, it becomes 
necessary to resort to more elaborate security measures. 
For example, a unit must perhaps prepare for three or more 
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alternate forms A, B and C rather than two. Equally 

important , students then should not witness the system of 
distribution to see who has the same test form. It has 
occasionally been the practice in some ELUs, for example , to 
distribute all test forms face down in advance and then to seat 
the students , telling them not to touch their tests booklets 
until instructed to do so, 

3 . Seating of Students 

27. An important security precaution for all major exam- 
nations vulnerable to compromise is the practice of not per- 
mitting students to sit where they wish. Students should enter 
the testing room one at a time and be seated by. the proctors. 
Ideally, teachers who know the students should be their proctors 
if at all possible, so that friends can not sit near each other. 
In particular, the best students in each group should be 
directed to the back of each testing room, where there will 

be no pressure on them to help their less able classmates. 

4 . Timing of Major Tests 

28. One of the best ways of preventing students from cheating 
on major examinations, especially on finals, is to time all 
parts fairly but realistically. Efficiency of language usage 
in all the language modes is a basic criterion of proficiency 
and timing of tests simply reflects this fundamental 
consideration. 

29. A test written to help all students finish everything 
is not being directed at the average majority; it is being, 
prepared at the lowest level of proficiency: those students 
most in danger of failing a course. In the event of such a 
test, the great majority of students find it so short, and 
usually easy, that they can leave the examination room prior 
to the end of the scheduled test time. 

30. Such a test usually leads to a serious imbalance of 
A and B grades vis-a-vis Cs, Ds and Fs . This is often so 
because, being too short and easy, it gives the great majority 
of students of average ability, at that level, too much time 
to (turtle and to carry cut pre-arranged "imtual assistance strategies. " 
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It is in no way challenging for the best students, who do not 
have to cheat anyway. Under such circumstances, mediocre 
students can qualify for exam scores and final grades they do 
not deserve, thus contributing to " grade inflation" and a 
consequent general lowering of academic standards. 

31. Timing can be informal (e.g., advisory) and can be 
communicated both in the test booklet and on the blackboard 
during the exam. Or it can be rigorously enforced for each 
part of a test, so that students are prohibited from looking at 
any test part other than that which they are instructed to be 
doing at a given time. , In the latter case, test parts must be 
printed on paper of a different color, so that teachers ^can see 
at a glance whether any students are looking at a part of the 
exam which tbey are, at that moment, not authorized to be doing. 

32. The claim that, during a course common achievement exam, 
especially a final, students should have a right to be able to 
review previously completed work is a strong and valid one. The 
needs of both students and testers can be accommodated (in the 
case of formal timing) if students are informed in advance that, 
at the end, they will be allowed a final, say, 10 or 15 minutes 
(maximum) to quickly skim all parts of their tests to review 
their answers for errors. They should r of course, be permitted 
to write at this time, finishing up whatever they wish. It is 
important that the provision of this formal review time not be 
abused by giving students more time than is needed by those 
already finished with all parts to skim their answers. This 
practice is reasonable, however, only if a test 1) has been 
formally timed, 2) is comprehensive, 3) is truly challenging 
throughout for the majority of students, and 4) is long enough 
both a) to keep all or virtually all students in the test 
room until the end of the scheduled test period and b) to 
keep the average majority of students busy writing virtually 
until the .final minutes of the test time allotted. 
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